Bug 2024522

Summary: [RFE] Provide a way to add a scsi **mpath** fencing device to a cluster without requiring a restart of all cluster resources
Product: Red Hat Enterprise Linux 9 Reporter: Tomas Jelinek <tojeline>
Component: pcsAssignee: Miroslav Lisik <mlisik>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: urgent Docs Contact: Steven J. Levine <slevine>
Priority: urgent    
Version: 9.0CC: cluster-maint, cluster-qe, idevat, kgaillot, kmalyjur, mlisik, mmazoure, mpospisi, nhostako, omular, sbradley, slevine, tojeline
Target Milestone: rcKeywords: FutureFeature, Triaged
Target Release: 9.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pcs-0.11.2-1.el9 Doc Type: Enhancement
Doc Text:
.`pcs` supports updating multipath SCSI devices without requiring a system restart You can now update multipath SCSI devices with the `pcs stonith update-scsi-devices` command. This command updates SCSI devices without causing a restart of other cluster resources running on the same node.
 Story Points: ---
Clone Of: 2023845 Environment:
Last Closed: 2022-11-15 09:48:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1872376, 1872378, 2023845    
Bug Blocks:    

Comment 2 Miroslav Lisik 2022-03-10 17:43:31 UTC 
Upstream commit: https://github.com/ClusterLabs/pcs/commit/bcf5526cb887be9987ae177ac494097f4532a949

Updated command:
pcs stonith update-scsi-devices

Test:
# pcs stonith update-scsi-devices <mpath-fence-device> set <device-path>...
# pcs stonith update-scsi-devices <mpath-fence-device> add <device-path>... remove <device-path>...

Devices of fence_mpath instance should be unfenced and updated and no resources should be restarted.
Comment 4 Miroslav Lisik 2022-05-19 17:17:06 UTC 
DevTestResults:

[root@r91-1 ~]# rpm -q pcs
pcs-0.11.2-1.el9.x86_64

### shortcuts for disk paths

[root@r91-1 ~]# for d in $mdisk{1..4}; do echo $d; done
/dev/disk/by-id/dm-uuid-mpath-36001405ae93fea8ee14438c843cb9e87
/dev/disk/by-id/dm-uuid-mpath-36001405b2eb733c07d34dbc87d393464
/dev/disk/by-id/dm-uuid-mpath-36001405eafcac8fb5004997b02bacbc7
/dev/disk/by-id/dm-uuid-mpath-360014053573a41da4a24ab7acde57014

### multipath configuration

[root@r91-1 ~]# multipath -ll
mpatha (36001405ae93fea8ee14438c843cb9e87) dm-0 LIO-ORG,r91-mdisk-01
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:0 sdd 8:48  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 15:0:0:0 sdf 8:80  active ready running
mpathb (36001405b2eb733c07d34dbc87d393464) dm-1 LIO-ORG,r91-mdisk-02
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:1 sdk 8:160 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 15:0:0:1 sdl 8:176 active ready running
mpathc (36001405eafcac8fb5004997b02bacbc7) dm-2 LIO-ORG,r91-mdisk-03
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:2 sdh 8:112 active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 15:0:0:2 sdj 8:144 active ready running
mpathd (360014053573a41da4a24ab7acde57014) dm-3 LIO-ORG,r91-mdisk-04
size=100M features='0' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 10:0:0:3 sdg 8:96  active ready running
`-+- policy='service-time 0' prio=50 status=enabled
  `- 15:0:0:3 sdi 8:128 active ready running

### list of reservation keys

[root@r91-1 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x5,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.

#### create fence-mpath stonith device and check the configuration

[root@r91-1 ~]# pcs stonith create mpath-fence-device fence_mpath devices="$mdisk1" pcmk_host_check="static-list" pcmk_host_list="r91-1 r91-2" pcmk_host_map="r91-1:1;r91-2:2" pcmk_host_argument="key" pcmk_reboot_action="off" meta provides="unfencing"
[root@r91-1 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-36001405ae93fea8ee14438c843cb9e87
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r91-1:1;r91-2:2
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s
[root@r91-1 ~]# pcs stonith
  * mpath-fence-device  (stonith:fence_mpath):   Started r91-1
[root@r91-1 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x9,    4 registered reservation keys follow:
    0x1
    0x1
    0x2
    0x2
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.
  PR generation=0x0,    0 registered reservation key.


### create some resources for restart detection checks

[root@r91-1 ~]# for i in $(seq -w 01 04); do pcs resource create d-$i ocf:pacemaker:Dummy; done
[root@r91-1 ~]# pcs resource
  * d-01        (ocf:pacemaker:Dummy):   Started r91-2
  * d-02        (ocf:pacemaker:Dummy):   Started r91-1
  * d-03        (ocf:pacemaker:Dummy):   Started r91-2
  * d-04        (ocf:pacemaker:Dummy):   Started r91-1

### update fence-mpath device configuration and check if the resources restarted

[root@r91-1 ~]# pcs stonith update-scsi-devices mpath-fence-device set $mdisk1 $mdisk2

[root@r91-1 ~]# journalctl -n 0 -f
May 16 19:22:16 r91-1 kernel: sd 15:0:0:1: reservation conflict
May 16 19:22:16 r91-1 kernel: sd 10:0:0:1: reservation conflict
May 16 19:22:17 r91-1 pacemaker-fenced[4262]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r91-2 ~]# journalctl -n 0 -f
May 16 19:22:16 r91-2 kernel: sd 7:0:0:1: reservation conflict
May 16 19:22:16 r91-2 kernel: sd 8:0:0:1: reservation conflict
May 16 19:22:17 r91-2 pacemaker-controld[3915]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 16 19:22:17 r91-2 pacemaker-fenced[3911]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 16 19:22:17 r91-2 pacemaker-schedulerd[3914]:  notice: Calculated transition 10, saving inputs in /var/lib/pacemaker/pengine/pe-input-29.bz2
May 16 19:22:17 r91-2 pacemaker-controld[3915]:  notice: Transition 10 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-29.bz2): Complete
May 16 19:22:17 r91-2 pacemaker-controld[3915]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r91-1 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x9,    4 registered reservation keys follow:
    0x1
    0x1
    0x2
    0x2
  PR generation=0x4,    4 registered reservation keys follow:
    0x1
    0x1
    0x2
    0x2


[root@r91-1 ~]# pcs stonith update-scsi-devices mpath-fence-device add $mdisk3 $mdisk4 remove $mdisk2
[root@r91-1 ~]# pcs stonith config
Resource: mpath-fence-device (class=stonith type=fence_mpath)
  Attributes: mpath-fence-device-instance_attributes
    devices=/dev/disk/by-id/dm-uuid-mpath-360014053573a41da4a24ab7acde57014,/dev/disk/by-id/dm-uuid-mpath-36001405ae93fea8ee14438c843cb9e87,/dev/disk/by-id/dm-uuid-mpath-36001405eafcac8fb5004997b02bacbc7
    pcmk_host_argument=key
    pcmk_host_check=static-list
    pcmk_host_list="r91-1 r91-2"
    pcmk_host_map=r91-1:1;r91-2:2
    pcmk_reboot_action=off
  Meta Attributes: mpath-fence-device-meta_attributes
    provides=unfencing
  Operations:
    monitor: mpath-fence-device-monitor-interval-60s
      interval=60s

[root@r91-1 ~]# journalctl -n 0 -f
May 16 19:32:37 r91-1 pacemaker-fenced[4262]:  notice: Added 'mpath-fence-device' to device list (1 active device)

[root@r91-2 ~]# journalctl -n 0 -f
May 16 19:32:37 r91-2 kernel: sd 7:0:0:3: reservation conflict
May 16 19:32:37 r91-2 pacemaker-controld[3915]:  notice: State transition S_IDLE -> S_POLICY_ENGINE
May 16 19:32:37 r91-2 pacemaker-fenced[3911]:  notice: Added 'mpath-fence-device' to device list (1 active device)
May 16 19:32:37 r91-2 pacemaker-schedulerd[3914]:  notice: Calculated transition 11, saving inputs in /var/lib/pacemaker/pengine/pe-input-30.bz2
May 16 19:32:37 r91-2 pacemaker-controld[3915]:  notice: Transition 11 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-30.bz2): Complete
May 16 19:32:37 r91-2 pacemaker-controld[3915]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE

[root@r91-1 ~]# for d in $mdisk{1..4}; do mpathpersist -i -k -d $d ; done
  PR generation=0x9,    4 registered reservation keys follow:
    0x1
    0x1
    0x2
    0x2
  PR generation=0x4,    4 registered reservation keys follow:
    0x1
    0x1
    0x2
    0x2
  PR generation=0x4,    4 registered reservation keys follow:
    0x1
    0x1
    0x2
    0x2
  PR generation=0x4,    4 registered reservation keys follow:
    0x1
    0x1
    0x2
    0x2
Comment 14 errata-xmlrpc 2022-11-15 09:48:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: pcs security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7935