RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1811072 - pacemaker should reject clones off fencing device resources
Summary: pacemaker should reject clones off fencing device resources
Keywords:
Status: CLOSED ERRATA
Alias: None
Deadline: 2021-12-06
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: pcs
Version: 9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 9.0
Assignee: Miroslav Lisik
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-06 14:38 UTC by Heinz Mauelshagen
Modified: 2022-05-17 12:22 UTC (History)
9 users (show)

Fixed In Version: pcs-0.11.1-7.el9
Doc Type: Bug Fix
Doc Text:
Cause: Commands `pcs resource clone/promotable` allows to clone stonith resources or groups containing stonith resources even if it is not recommended or necessary. Consequence: It is confusing that commands `pcs resource clone/promotable` allows to clone stonith resources though it is not recommended. Fix: Pcs commands `pcs resource clone/promotable` reject to clone stonith resources or groups containing stonith resources with an error but allows to override it with the '--force' option. Result: Pcs commands `pcs resource clone/promotable` reject to clone stonith resources or resource group containing stonith resources by default.
Clone Of:
Environment:
Last Closed: 2022-05-17 12:19:34 UTC
Type: ---
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
proposed fix + tests (12.47 KB, patch)
2021-10-11 16:09 UTC, Miroslav Lisik 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CLUSTERQE-5147 0 None None None 2021-10-25 19:43:55 UTC
Red Hat Issue Tracker CLUSTERQE-5189 0 None None None 2021-10-27 13:07:46 UTC
Red Hat Product Errata RHBA-2022:2290 0 None None None 2022-05-17 12:19:56 UTC

Description Heinz Mauelshagen 2020-03-06 14:38:10 UTC 
Description of problem:

Fencing devices (e.g. fence_ipmlan) are created as a resource which will get started on one
cluster node and eventually get moved manually or by cluster resource management.

Though it is not recommended/supported to clone such fencing resource,
it is possible and my cause harm (not tested).

Please think about rejecting 'pcs resource clone '.
Comment 1 Ken Gaillot 2020-03-06 15:43:12 UTC 
I investigated more and realized I was wrong about cloning stonith devices. Stonith devices *can* be cloned as anonymous clones (the default type), just not as unique clones (where more than one copy can run on a single node).

The only benefit from cloning a stonith device would be to ensure that the device is reachable from every node on a regular basis.

However I think more likely than not, someone trying to clone a stonith device doesn't realize that any node can use a fence device (unless specifically banned), regardless of whether the fence device is running on that node, or even running at all.

Addressing such user-end concerns is handled by pcs, so I'm reassigning this bz to that component. The "pcs stonith create" command already doesn't offer the option to create a clone, so I think the only suggestion would be that "pcs resource clone" reject stonith devices if --force is not given, with an explanation that a node can use the device even if it's not running there.
Comment 7 RHEL Program Management 2021-09-06 07:27:07 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Comment 9 Miroslav Lisik 2021-10-11 16:09:18 UTC 
Created attachment 1831901 [details]
proposed fix + tests

Updated commands:
* pcs resource clone
* pcs resource promotable

Test:
* try to clone a stonith resource or group containing stonith resources: pcs commands exit with an error message
* try to clone a stonith resource or group containing stonith resources with --force option: pcs clones resource or group with warning message
Comment 11 Miroslav Lisik 2021-11-03 15:59:04 UTC 
DevTestResults:

[root@r90-node-01 ~]# rpm -q pcs
pcs-0.11.1-4.el9.x86_64

[root@r90-node-01 ~]# pcs stonith
  * fence-r90-node-01   (stonith:fence_xvm):     Started r90-node-01
  * fence-r90-node-02   (stonith:fence_xvm):     Started r90-node-02

[root@r90-node-01 ~]# pcs stonith
  * fence-r90-node-01   (stonith:fence_xvm):     Started r90-node-01
    * fence-r90-node-02 (stonith:fence_xvm):     Started r90-node-02
[root@r90-node-01 ~]# pcs resource config
 Group: G
  Resource: fence-r90-node-02 (class=stonith type=fence_xvm)
   Attributes: pcmk_host_check=static-list pcmk_host_list=r90-node-02 pcmk_host_map=r90-node-02:r90-node-02
   Operations: monitor interval=60s (fence-r90-node-02-monitor-interval-60s)
  Resource: d-01 (class=ocf provider=pacemaker type=Dummy)
   Operations: migrate_from interval=0s timeout=20s (d-01-migrate_from-interval-0s)
               migrate_to interval=0s timeout=20s (d-01-migrate_to-interval-0s)
               monitor interval=10s timeout=20s (d-01-monitor-interval-10s)
               reload interval=0s timeout=20s (d-01-reload-interval-0s)
               reload-agent interval=0s timeout=20s (d-01-reload-agent-interval-0s)
               start interval=0s timeout=20s (d-01-start-interval-0s)
               stop interval=0s timeout=20s (d-01-stop-interval-0s)

### cloning a resource

[root@r90-node-01 ~]# pcs resource clone fence-r90-node-01
Error: No need to clone stonith resource 'fence-r90-node-01', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not, use --force to override
[root@r90-node-01 ~]# echo $?
1
[root@r90-node-01 ~]# pcs resource promotable fence-r90-node-01
Error: No need to clone stonith resource 'fence-r90-node-01', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not, use --force to override
[root@r90-node-01 ~]# echo $?
1

### cloning a resource (with force)

[root@r90-node-01 ~]# pcs resource clone fence-r90-node-01 --force
Warning: No need to clone stonith resource 'fence-r90-node-01', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not
[root@r90-node-01 ~]# echo $?
0
[root@r90-node-01 ~]# pcs resource
  * Resource Group: G:
    * d-01      (ocf:pacemaker:Dummy):   Started r90-node-02
  * Clone Set: fence-r90-node-01-clone [fence-r90-node-01]:
    * Started: [ r90-node-01 r90-node-02 ]


### cloning a group with stonith

[root@r90-node-01 ~]# pcs resource clone G
Error: Group 'G' contains stonith resource. No need to clone stonith resource 'fence-r90-node-02', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not, use --force to override
[root@r90-node-01 ~]# echo $?
1

### cloning a group with stonith (with force)

[root@r90-node-01 ~]# pcs resource clone G --force
Warning: Group 'G' contains stonith resource. No need to clone stonith resource 'fence-r90-node-02', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not
[root@r90-node-01 ~]# echo $?
0
[root@r90-node-01 ~]# pcs resource
  * Clone Set: fence-r90-node-01-clone [fence-r90-node-01]:
    * Started: [ r90-node-01 r90-node-02 ]
  * Clone Set: G-clone [G]:
    * Started: [ r90-node-01 r90-node-02 ]
Comment 18 Ondrej Mular 2021-12-13 12:25:51 UTC 
(In reply to svalasti from comment #15)
> ...snip...
> ## NEED FIX
> ================
> ## Set up stonith device as promotable clone.
> ...snip...
> 
> > Stonith device is not shown in pcs status and pcs stonith.
> > It is also not possible to disable this stonith device. However, this device can still be removed.
> 
This is not a fault of pcs, as it takes this output directly from pacemaker (`crm_mon --one-shot --inactive`). Actually I believe this is a misconfiguration (based on output of `pcs cluster verify --full`) therefore pacemaker is not including it in the status output.
> 
> ## Adding stonith device to cloned group.
> ...snip...
> > It is still possible to add stonith device to the cloned group of non-stonith resources.
It turns out that a proper fix for cloned groups is a much more complex task that we originally anticipated. Therefore we will revert the check for groups to stay consistent. This case will be addressed as part of bz#1301204 (which will be reopened) in a future release.
Comment 19 Tomas Jelinek 2021-12-13 12:37:37 UTC 
(In reply to svalasti from comment #15)
> ## NEED FIX
> ================
> [root@virt-155 ~]# pcs resource disable fence-virt-155
> Error: bundle/clone/group/resource 'fence-virt-155' does not exist
> Error: Errors have occurred, therefore pcs is unable to continue
> [root@virt-155 ~]# echo $?
>   1
> > It is also not possible to disable this stonith device.

Have you tried 'pcs stonith disable'?
Comment 20 Ondrej Mular 2021-12-14 15:26:07 UTC 
Additional patch reverting previous change:
https://github.com/ClusterLabs/pcs/commit/f245bf6bb3370883dc7831c49184eb336f9ef096
Comment 21 svalasti 2021-12-16 12:22:53 UTC 
(In reply to Tomas Jelinek from comment #19)
> (In reply to svalasti from comment #15)
> > ## NEED FIX
> > ================
> > [root@virt-155 ~]# pcs resource disable fence-virt-155
> > Error: bundle/clone/group/resource 'fence-virt-155' does not exist
> > Error: Errors have occurred, therefore pcs is unable to continue
> > [root@virt-155 ~]# echo $?
> >   1
> > > It is also not possible to disable this stonith device.
> 
> Have you tried 'pcs stonith disable'?

Yes, I tried 'pcs stonith disable', the results are same as 'pcs resource disable'.

[root@virt-011 ~]# rpm -q pcs
pcs-0.11.1-6.el9.x86_64
[root@virt-011 ~]# pcs stonith
  * fence-virt-011	(stonith:fence_xvm):	 Started virt-011
  * fence-virt-012	(stonith:fence_xvm):	 Started virt-012
[root@virt-011 ~]# pcs resource promotable fence-virt-011 --force
Warning: No need to clone stonith resource 'fence-virt-011', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not
[root@virt-011 ~]# echo $?
0
[root@virt-011 ~]# pcs stonith
  * fence-virt-012	(stonith:fence_xvm):	 Started virt-012
[root@virt-011 ~]# pcs stonith disable fence-virt-011
Error: bundle/clone/group/resource 'fence-virt-011' does not exist
Error: Errors have occurred, therefore pcs is unable to continue
[root@virt-011 ~]# echo $?
1
[root@virt-011 ~]# pcs resource remove fence-virt-011
Deleting Resource - fence-virt-011
[root@virt-011 ~]# echo $?
Comment 22 Tomas Jelinek 2021-12-16 13:57:29 UTC 
(In reply to svalasti from comment #21)
> (In reply to Tomas Jelinek from comment #19)
> > (In reply to svalasti from comment #15)
> > > ## NEED FIX
> > > ================
> > > [root@virt-155 ~]# pcs resource disable fence-virt-155
> > > Error: bundle/clone/group/resource 'fence-virt-155' does not exist
> > > Error: Errors have occurred, therefore pcs is unable to continue
> > > [root@virt-155 ~]# echo $?
> > >   1
> > > > It is also not possible to disable this stonith device.
> > 
> > Have you tried 'pcs stonith disable'?
> 
> Yes, I tried 'pcs stonith disable', the results are same as 'pcs resource disable'.

This is what's going on:
Pcs loads cluster status XML and searches for fence-virt-155 resource in it to figure out if the resource is managed or unmanaged. Since stonith resources cannot be made promotable, the resource is not present in the status at all. That's why pcs fails. The solution is to implement a fix for bz1493416.

Since you needed to --force the command to make the stonith resource promotable, I would not call it a bug. When you use --force, you acknowledge that the resulting configuration may not be working.
Comment 23 Miroslav Lisik 2021-12-16 16:21:35 UTC 
DevTestResults:

[root@r90-node-01 ~]# rpm -q pcs
pcs-0.11.1-7.el9.x86_64


### configured stonith resource in a group

[root@r90-node-01 ~]# pcs stonith
  * fence-r90-node-01   (stonith:fence_xvm):     Started r90-node-01
    * fence-r90-node-02 (stonith:fence_xvm):     Started r90-node-02
[root@r90-node-01 ~]# pcs resource
  * Resource Group: GroupWithStonith:
    * d-01      (ocf:pacemaker:Dummy):   Started r90-node-02
[root@r90-node-01 ~]# pcs resource config
 Group: GroupWithStonith
  Resource: fence-r90-node-02 (class=stonith type=fence_xvm)
   Attributes: pcmk_host_check=static-list pcmk_host_list=r90-node-02 pcmk_host_map=r90-node-02:r90-node-02
   Operations: monitor interval=60s (fence-r90-node-02-monitor-interval-60s)
  Resource: d-01 (class=ocf provider=pacemaker type=Dummy)
   Operations: migrate_from interval=0s timeout=20s (d-01-migrate_from-interval-0s)
               migrate_to interval=0s timeout=20s (d-01-migrate_to-interval-0s)
               monitor interval=10s timeout=20s (d-01-monitor-interval-10s)
               reload interval=0s timeout=20s (d-01-reload-interval-0s)
               reload-agent interval=0s timeout=20s (d-01-reload-agent-interval-0s)
               start interval=0s timeout=20s (d-01-start-interval-0s)
               stop interval=0s timeout=20s (d-01-stop-interval-0s)

### clone a group with a stonith resource

[root@r90-node-01 ~]# pcs resource clone GroupWithStonith
[root@r90-node-01 ~]# pcs resource
  * Clone Set: GroupWithStonith-clone [GroupWithStonith]:
    * Started: [ r90-node-01 r90-node-02 ]
[root@r90-node-01 ~]# pcs resource config
 Clone: GroupWithStonith-clone
  Group: GroupWithStonith
   Resource: fence-r90-node-02 (class=stonith type=fence_xvm)
    Attributes: pcmk_host_check=static-list pcmk_host_list=r90-node-02 pcmk_host_map=r90-node-02:r90-node-02
    Operations: monitor interval=60s (fence-r90-node-02-monitor-interval-60s)
   Resource: d-01 (class=ocf provider=pacemaker type=Dummy)
    Operations: migrate_from interval=0s timeout=20s (d-01-migrate_from-interval-0s)
                migrate_to interval=0s timeout=20s (d-01-migrate_to-interval-0s)
                monitor interval=10s timeout=20s (d-01-monitor-interval-10s)
                reload interval=0s timeout=20s (d-01-reload-interval-0s)
                reload-agent interval=0s timeout=20s (d-01-reload-agent-interval-0s)
                start interval=0s timeout=20s (d-01-start-interval-0s)
                stop interval=0s timeout=20s (d-01-stop-interval-0s)

It is possible to clone a group with a stonith resource without --force option.
Comment 24 svalasti 2022-01-17 13:12:55 UTC 
[root@virt-491 ~]# rpm -q pcs
pcs-0.11.1-7.el9.x86_64

## Cloning group with a stonith device
[root@virt-491 ~]# pcs status
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 virt-493 ]

Full List of Resources:
  * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
  * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
  * d1	(ocf:pacemaker:Dummy):	 Started virt-491
  * d2	(ocf:pacemaker:Dummy):	 Started virt-493
...

[root@virt-491 ~]# pcs resource group add g1 d1 d2 fence-virt-491
[root@virt-491 ~]# echo $?
0
[root@virt-491 ~]# pcs resource clone g1
[root@virt-491 ~]# pcs status --full
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 (1) virt-493 (2) ]

Full List of Resources:
  * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
  * Clone Set: g1-clone [g1]:
    * Resource Group: g1:0:
      * d1	(ocf:pacemaker:Dummy):	 Started virt-493
      * d2	(ocf:pacemaker:Dummy):	 Started virt-493
      * fence-virt-491	(stonith:fence_xvm):	 Started virt-493
    * Resource Group: g1:1:
      * d1	(ocf:pacemaker:Dummy):	 Started virt-491
      * d2	(ocf:pacemaker:Dummy):	 Started virt-491
      * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
...
> OK

## Adding a stonith device to the cloned group
[root@virt-491 ~]# pcs status
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 virt-493 ]

Full List of Resources:
  * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
  * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
  * d1	(ocf:pacemaker:Dummy):	 Started virt-491
  * d2	(ocf:pacemaker:Dummy):	 Started virt-493
...

[root@virt-491 ~]# pcs resource group add g2 d1 d2
[root@virt-491 ~]# pcs resource clone g2
[root@virt-491 ~]# pcs resource group add g2 fence-virt-491
[root@virt-491 ~]# pcs status --full
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 (1) virt-493 (2) ]

Full List of Resources:
  * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
  * Clone Set: g2-clone [g2]:
    * Resource Group: g2:0:
      * d1	(ocf:pacemaker:Dummy):	 Started virt-493
      * d2	(ocf:pacemaker:Dummy):	 Started virt-493
      * fence-virt-491	(stonith:fence_xvm):	 Started virt-493
    * Resource Group: g2:1:
      * d1	(ocf:pacemaker:Dummy):	 Started virt-491
      * d2	(ocf:pacemaker:Dummy):	 Started virt-491
      * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
...
> OK

## Creating and clonning group consisting only of stonith devices
[root@virt-491 ~]# pcs status
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 virt-493 ]

Full List of Resources:
  * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
  * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
...
[root@virt-491 ~]# pcs resource group add g3 fence-virt-491 fence-virt-493
[root@virt-491 ~]# echo $?
0
[root@virt-491 ~]# pcs resource clone g3
[root@virt-491 ~]# echo $?
0
[root@virt-491 ~]# pcs status --full
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 (1) virt-493 (2) ]

Full List of Resources:
  * Clone Set: g3-clone [g3]:
    * Resource Group: g3:0:
      * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
      * fence-virt-493	(stonith:fence_xvm):	 Started virt-491
    * Resource Group: g3:1:
      * fence-virt-491	(stonith:fence_xvm):	 Started virt-493
      * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
...
> OK

## Trying to clone stonith device.

[root@virt-491 ~]# pcs status --full
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 (1) virt-493 (2) ]

Full List of Resources:
  * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
  * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
...
[root@virt-491 ~]# pcs resource clone fence-virt-491
Error: No need to clone stonith resource 'fence-virt-491', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not, use --force to override
[root@virt-491 ~]# echo $?
1
[root@virt-491 ~]# pcs resource clone fence-virt-491 --force
Warning: No need to clone stonith resource 'fence-virt-491', any node can use a stonith resource (unless specifically banned) regardless of whether the stonith resource is running on that node or not
[root@virt-491 ~]# echo $?
0
[root@virt-491 ~]# pcs status --full
Cluster name: STSRHTS23947
...
Node List:
  * Online: [ virt-491 (1) virt-493 (2) ]

Full List of Resources:
  * fence-virt-493	(stonith:fence_xvm):	 Started virt-493
  * Clone Set: fence-virt-491-clone [fence-virt-491]:
    * fence-virt-491	(stonith:fence_xvm):	 Started virt-491
    * fence-virt-491	(stonith:fence_xvm):	 Started virt-493
...
> The rest of the stonith device`s behavior has remained same since the last verification.
Comment 26 errata-xmlrpc 2022-05-17 12:19:34 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: pcs), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2290
Note You need to log in before you can comment on or make changes to this bug.