Bug 2023845

Summary: [RFE] Provide a way to add a scsi **mpath** fencing device to a cluster without requiring a restart of all cluster resources
Product: Red Hat Enterprise Linux 8 Reporter: Chris Feist <cfeist>
Component: pcsAssignee: Miroslav Lisik <mlisik>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: urgent Docs Contact: Steven J. Levine <slevine>
Priority: urgent    
Version: 8.5CC: cluster-maint, cluster-qe, idevat, kgaillot, kmalyjur, mlisik, mmazoure, mpospisi, nhostako, omular, sbradley, slevine, tojeline
Target Milestone: rcKeywords: FutureFeature, Triaged
Target Release: 8.7Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pcs-0.10.13-1.el8 Doc Type: Enhancement
Doc Text:
.`pcs` command-line supports updating multipath SCSI devices without requiring a system restart You can now update multipath SCSI devices with the `pcs stonith update-scsi-devices` command. This command updates SCSI devices without causing a restart of other cluster resources running on the same node.
 Story Points: ---
Clone Of: 1872378
: 2024522 (view as bug list) Environment:
Last Closed: 2022-11-08 09:12:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1872376, 1872378    
Bug Blocks: 2024522    

Comment 5 Miroslav Lisik 2022-03-17 09:33:19 UTC 
Upstream commit: https://github.com/ClusterLabs/pcs/commit/7eb7d3c2c57c6292c470c13af262afb9368f564b

Updated command:
pcs stonith update-scsi-devices

Test:
# pcs stonith update-scsi-devices <mpath-fence-device> set <device-path>...
# pcs stonith update-scsi-devices <mpath-fence-device> add <device-path>... remove <device-path>...

Devices of fence_mpath instance should be unfenced and updated and no resources should be restarted.
Comment 7 Miroslav Lisik 2022-05-26 08:49:21 UTC 
DevTestResults:

[root@r8-node-01 ~]# rpm -q pcs
pcs-0.10.13-1.el8.x86_64

### shortcuts for disk paths

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do echo $d; done
/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb
/dev/disk/by-id/dm-uuid-mpath-36001405df3f8751c94649d3ae3f11d30
/dev/disk/by-id/dm-uuid-mpath-36001405eafb08de17a540bc9a97dedf3
/dev/disk/by-id/dm-uuid-mpath-360014054f2782eef9394e7aa72eea7e3

### multipath configuration

[root@r8-node-01 ~]# multipath -ll
mpatha (3600140526d1927596fd4577958a15dcb) dm-0 LIO-ORG,r8-mdisk-01
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:0 sdi 8:128 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:0  sda 8:0   active ready running
mpathb (36001405df3f8751c94649d3ae3f11d30) dm-1 LIO-ORG,r8-mdisk-02
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:1 sdl 8:176 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 6:0:0:1  sdd 8:48  active ready running
mpathc (36001405eafb08de17a540bc9a97dedf3) dm-2 LIO-ORG,r8-mdisk-03
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:2  sdc 8:32  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 10:0:0:2 sdk 8:160 active ready running
mpathd (360014054f2782eef9394e7aa72eea7e3) dm-3 LIO-ORG,r8-mdisk-04
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 6:0:0:3  sdb 8:16  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 10:0:0:3 sdj 8:144 active ready running

### list of reservation keys

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.

#### create fence-mpath stonith device and check the configuration

[root@r8-node-01 ~]# pcs stonith create mpath-fence-device fence_mpath devices="$mdisk1" pcmk_host_check="static-list" pcmk_host_list="r91-1 r91-2" pcmk_host_map="r8-node-01:1;r8-node-02:2;r8-node-03:3" pcmk_host_argument="key" pcmk_reboot_action="off"
[root@r8-node-01 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r8-node-01:1;r8-node-02:2;r8-node-03:3
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s
[root@r8-node-01 ~]# pcs stonith
  * mpath-fence-device  (stonith:fence_mpath):   Started r8-node-02
[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.

### create some resources for restart detection checks

[root@r8-node-01 ~]# for i in $(seq -w 01 04); do pcs resource create d-$i ocf:pacemaker:Dummy; done
[root@r8-node-01 ~]# pcs resource
  * d-01        (ocf::pacemaker:Dummy):  Started r8-node-01
  * d-02        (ocf::pacemaker:Dummy):  Started r8-node-02
  * d-03        (ocf::pacemaker:Dummy):  Started r8-node-03
  * d-04        (ocf::pacemaker:Dummy):  Started r8-node-01


### update fence-mpath device configuration and check if the resources restarted

1) Set syntax

[root@r8-node-01 ~]# pcs stonith update-scsi-devices mpath-fence-device set $mdisk1 $mdisk2

[root@r8-node-01 ~]#
[root@r8-node-01 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:20 CEST. --
May 25 15:25:39 r8-node-01 pacemaker-fenced[97569]:  notice: Added 'mpath-fence-device' to device list (1 active device)

root@r8-node-02 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:23 CEST. --
May 25 15:25:38 r8-node-02 kernel: sd 6:0:0:1: reservation conflict
May 25 15:25:39 r8-node-02 pacemaker-fenced[88581]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-03 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:31 CEST. --
May 25 15:25:38 r8-node-03 kernel: sd 9:0:0:1: reservation conflict
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 25 15:25:39 r8-node-03 pacemaker-fenced[91327]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 25 15:25:39 r8-node-03 pacemaker-schedulerd[91330]:  notice: Calculated transition 28, saving inputs in /var/lib/pacemaker/pengine/pe-input-1560.bz2
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: Transition 28 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1560.bz2): Complete
May 25 15:25:39 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x1
    0x1
    0x3
    0x3
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.


2) Add/Remove syntax

[root@r8-node-01 ~]# pcs stonith update-scsi-devices mpath-fence-device add $mdisk3 $mdisk4 remove $mdisk2
[root@r8-node-01 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-3600140526d1927596fd4577958a15dcb,/dev/disk/by-id/dm-uuid-mpath-360014054f2782eef9394e7aa72eea7e3,/dev/disk/by-id/dm-uuid-mpath-36001405eafb08de17a540bc9a97dedf3
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r8-node-01:1;r8-node-02:2;r8-node-03:3
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s

[root@r8-node-01 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:20 CEST. --
May 25 16:46:21 r8-node-01 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 10:0:0:3: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 10:0:0:2: reservation conflict
May 25 16:46:21 r8-node-01 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-01 pacemaker-fenced[97569]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-02 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:23 CEST. --
May 25 16:46:21 r8-node-02 kernel: sd 7:0:0:3: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 7:0:0:2: reservation conflict
May 25 16:46:21 r8-node-02 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-02 pacemaker-fenced[88581]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r8-node-03 ~]# journalctl -n 0 -f
-- Logs begin at Mon 2022-05-23 14:32:31 CEST. --
May 25 16:46:21 r8-node-03 kernel: sd 9:0:0:3: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 6:0:0:3: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 9:0:0:2: reservation conflict
May 25 16:46:21 r8-node-03 kernel: sd 6:0:0:2: reservation conflict
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 25 16:46:22 r8-node-03 pacemaker-fenced[91327]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 25 16:46:22 r8-node-03 pacemaker-schedulerd[91330]:  notice: Calculated transition 35, saving inputs in /var/lib/pacemaker/pengine/pe-input-1563.bz2
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: Transition 35 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1563.bz2): Complete
May 25 16:46:22 r8-node-03 pacemaker-controld[91331]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r8-node-01 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x6,    6 registered reservation keys follow:
    0x1
    0x1
    0x3
    0x3
    0x2
    0x2
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x1
    0x1
    0x3
    0x3
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x3
    0x3
    0x1
    0x1
  PR generation=0x6,    6 registered reservation keys follow:
    0x2
    0x2
    0x3
    0x3
    0x1
    0x1
Comment 19 errata-xmlrpc 2022-11-08 09:12:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: pcs security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7447