+++ This bug was initially created as a clone of Bug #1578789 +++

Description of problem:
* moving and banning an inner bundle resource
  * pcs should exit with an error saying moving / banning inner bundle resources is not allowed
* moving / banning a pacemaker internal bundle resource
  * works OK, pcs says the resource doesn't exist (it's not in the CIB)
* moving and banning a bundle resource
  * does not currently work in crm_resource even for 1-instance bundles:
    Resource 'DummyBundle' not moved: active in 2 locations.
* moving and banning a clone resource
  * works in crm_resource for 1-instance clones for both a clone and its inner resource
  * for multi-instance clones, moving a resource in a clone creates a constraint,  whereas moving a clone resource exits with an error
  * pcs should not allow moving / banning a resource in a clone, then it can simply run crm_resource every time and based on its output inform a user if a clone could or couldn't be moved / banned
  * another option is to automatically switch to moving / banning a bundle / clone if the user tries to move / ban a resource in the bundle / clone


Version-Release number of selected component (if applicable):
pcs-0.9.162-5.el7.x86_64


How reproducible:
always, easily


Steps to Reproduce:

dummy0 is a resource inside DummyBundle bundle with replicas=1
--------------------------------------------------------------

[root@virt-143 ~]# pcs resource move dummy0
Warning: Creating location constraint cli-ban-dummy0-on-DummyBundle-0 with a score of -INFINITY for resource dummy0 on node DummyBundle-0.
This will prevent dummy0 from running on DummyBundle-0 until the constraint is removed. This will be the case even if DummyBundle-0 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy0
    Disabled on: DummyBundle-0 (score:-INFINITY) (role: Started) (id:cli-ban-dummy0-on-DummyBundle-0)

[root@virt-143 ~]# crm_resource --resource dummy0 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy0-on-DummyBundle-0' with a score of -INFINITY for resource dummy0 on DummyBundle-0.
	This will prevent dummy0 from running on DummyBundle-0 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if DummyBundle-0 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy0
    Disabled on: DummyBundle-0 (score:-INFINITY) (role: Started) (id:cli-ban-dummy0-on-DummyBundle-0)

> Constraint has been created, but resource WAS NOT moved. The reason is obvious as DummyBundle-0 is not an actual node. This is a possible place for improvement.


[root@virt-143 ~]# pcs resource move DummyBundle-0
Error: DummyBundle-0 is not a valid resource

[root@virt-143 ~]# crm_resource --resource DummyBundle-0 --move
WARNING: Creating rsc_location constraint 'cli-ban-DummyBundle-0-on-virt-143' with a score of -INFINITY for resource DummyBundle-0 on virt-143.
	This will prevent DummyBundle-0 from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-143 is the last node in the cluster
	This message can be disabled with --quiet
Error performing operation: Update does not conform to the configured schema

> Neither pcs not crm_resource is able to target the DummyBundle-0 pseudo resource which is expected.


[root@virt-143 ~]# pcs resource move DummyBundle
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource DummyBundle --move
Resource 'DummyBundle' not moved: active in 2 locations.
You can prevent 'DummyBundle' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

> In this situation pcs simply shouts for trying to move a clone, while crm_resource gives us something else - a misinformation about the number of locations being active. There should only be one active due to replicas=1. Let's see how well the pure clones will do:


dummy1 is a clone with clone-max=1
----------------------------------

[root@virt-143 ~]# pcs resource move dummy1
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource dummy1 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-145' with a score of -INFINITY for resource dummy1 on virt-145.
	This will prevent dummy1 from running on virt-145 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-145 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Disabled on: virt-145 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-145)

> A clone with single instance was refused to move by pcs, but actually passed and WAS moved with crm_resource. There might be a room for improvement in pcs.


[root@virt-143 ~]# pcs resource move dummy1-clone
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource dummy1-clone --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-clone-on-virt-143' with a score of -INFINITY for resource dummy1-clone on virt-143.
	This will prevent dummy1-clone from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-143 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1-clone
    Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-clone-on-virt-143)

> When targeting clone ID, once again a clone with single instance was refused to move by pcs, but passed and WAS moved with crm_resource.



dummy1 is a clone with clone-max=2
----------------------------------

[root@virt-143 ~]# pcs resource move dummy1
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource dummy1 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-143' with a score of -INFINITY for resource dummy1 on virt-143.
	This will prevent dummy1 from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-143 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)


[root@virt-143 ~]# crm_resource --resource dummy1 --move
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-145' with a score of -INFINITY for resource dummy1 on virt-145.
	This will prevent dummy1 from running on virt-145 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-145 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)
    Disabled on: virt-145 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-145)

> With more than one clone instance crm_resource does surprisingly perform a constraint-based move by moving ONE of the instance. Even if ran twice in a row.


[root@virt-143 ~]# pcs resource move dummy1-clone
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource dummy1-clone --move
Resource 'dummy1-clone' not moved: active in 2 locations.
You can prevent 'dummy1-clone' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

> Here with clone ID and two instances crm_resource won't do us a favor anymore. Pcs keeps printing the same error as before.


So, the conclusion for 'resource move':
 - pcs is not really up in sync with crm_resource behavior in terms of being able to move a single clone instance
 - support for moving a bundle in crm_resource is not on par with being able to move a clone, at least in case of just one instance being present due to:
     - 2 active locations being improperly reported 
     - the move constraint of a primitive is incorrectly targeted to a non-existent DummyBundle-0 node.

It is worth pointing out that creating a pure location constraint for DummyBundle will move the only existing bundle instance to the desired node as one would expect. I yet have to test the behavior of 'resource ban'.




Performing tests of move with node specification. 

dummy0 is a resource inside DummyBundle bundle with replicas=1
--------------------------------------------------------------

[root@virt-143 ~]# pcs resource move dummy0 virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy0
    Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy0)

[root@virt-143 ~]# crm_resource --resource dummy0 --move --node virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy0
    Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy0)

> Constraint has been created, but resource WAS NOT moved.


[root@virt-143 ~]# pcs resource move DummyBundle-0 virt-145
Error: DummyBundle-0 is not a valid resource

[root@virt-143 ~]# crm_resource --resource DummyBundle-0 --move --node virt-145
Error performing operation: Update does not conform to the configured schema

> Neither pcs not crm_resource is able to target the DummyBundle-0 pseudo resource which is expected.


[root@virt-143 ~]# pcs resource move DummyBundle virt-145
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource DummyBundle --move --node virt-145
Resource 'DummyBundle' not moved: active on multiple nodes
Error performing operation: Invalid argument

> Just like in the previous case the crm_resource gives us a misinformation about the number of locations being active. There should only be one active due to replicas=1.


dummy1 is a clone with clone-max=1
----------------------------------

[root@virt-143 ~]# pcs resource move dummy1 virt-145
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource dummy1 --move --node virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1)

> A clone with single instance was refused to move by pcs, but actually passed and WAS moved with crm_resource. There might be a room for improvement in pcs.


[root@virt-143 ~]# pcs resource move dummy1-clone virt-146
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource dummy1-clone --move --node virt-146
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1-clone
    Enabled on: virt-146 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1-clone)

> When targeting clone ID, once again a clone with single instance was refused to move by pcs, but passed and WAS moved with crm_resource.


dummy1 is a clone with clone-max=2
----------------------------------

[root@virt-143 ~]# pcs resource move dummy1 virt-145
Error: cannot move cloned resources

[root@virt-143 ~]# crm_resource --resource dummy1 --move --node virt-145
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Enabled on: virt-145 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1)

> With more than one clone instance crm_resource does perform a constraint-based move by moving ONE of the instance.


[root@virt-143 ~]# pcs resource move dummy1-clone virt-146
Error: cannot move cloned resources
[root@virt-143 ~]# 
[root@virt-143 ~]# crm_resource --resource dummy1-clone --move --node virt-146
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1-clone
    Enabled on: virt-146 (score:INFINITY) (role: Started) (id:cli-prefer-dummy1-clone)

> Here with clone ID and two instances crm_resource won't do us a favor anymore. Pcs keeps printing the same error as before.


The conclusion for 'resource move' with node specification is pretty much aligned with the previous one:
 - pcs is not really up in sync with crm_resource behavior in terms of being able to move a single clone instance
 - support for moving a bundle in crm_resource is not on par with being able to move a clone, at least in case of just one instance being present due to:
     - 2 active locations being improperly reported 
     - the move constraint of a primitive is incorrectly targeted to a non-existent DummyBundle-0 node.




Performing tests of ban with crm_resource is pretty much identical to simple move (with no node specification), so I am going to mention pcs differences only:

dummy0 is a resource inside DummyBundle bundle with replicas=1
--------------------------------------------------------------

[root@virt-143 ~]# pcs resource ban DummyBundle
Error: error moving/banning/clearing resource
Resource 'DummyBundle' not moved: active in 2 locations.
You can prevent 'DummyBundle' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

[root@virt-143 ~]# crm_resource --resource DummyBundle --ban
Resource 'DummyBundle' not moved: active in 2 locations.
You can prevent 'DummyBundle' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

> The difference is in pcs output when targetting DummyBundle which no longer complains about a clone but instead passes the crm_resource message as is. We'll need a more user friendly message in pcs.


dummy1 is a clone with clone-max=1
----------------------------------

[root@virt-143 ~]# pcs resource ban dummy1
Warning: Creating location constraint cli-ban-dummy1-on-virt-143 with a score of -INFINITY for resource dummy1 on node virt-143.
This will prevent dummy1 from running on virt-143 until the constraint is removed. This will be the case even if virt-143 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)

[root@virt-143 ~]# crm_resource --resource dummy1 --ban
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-144' with a score of -INFINITY for resource dummy1 on virt-144.
	This will prevent dummy1 from running on virt-144 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-144 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Disabled on: virt-144 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-144)

> The resource WAS moved in both cases so pcs and crm_resource are in sync regarding behavior.


[root@virt-143 ~]# pcs resource ban dummy1-clone
Warning: Creating location constraint cli-ban-dummy1-clone-on-virt-143 with a score of -INFINITY for resource dummy1-clone on node virt-143.
This will prevent dummy1-clone from running on virt-143 until the constraint is removed. This will be the case even if virt-143 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1-clone
    Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-clone-on-virt-143)

[root@virt-143 ~]# crm_resource --resource dummy1-clone --ban
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-clone-on-virt-144' with a score of -INFINITY for resource dummy1-clone on virt-144.
	This will prevent dummy1-clone from running on virt-144 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-144 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1-clone
    Disabled on: virt-144 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-clone-on-virt-144)

> Again the resource WAS moved in both cases so pcs and crm_resource are in sync regarding behavior.



dummy1 is a clone with clone-max=2
----------------------------------

[root@virt-143 ~]# pcs resource ban dummy1
Warning: Creating location constraint cli-ban-dummy1-on-virt-145 with a score of -INFINITY for resource dummy1 on node virt-145.
This will prevent dummy1 from running on virt-145 until the constraint is removed. This will be the case even if virt-145 is the last node in the cluster.
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Disabled on: virt-145 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-145)

[root@virt-143 ~]# crm_resource --resource dummy1 --ban
WARNING: Creating rsc_location constraint 'cli-ban-dummy1-on-virt-143' with a score of -INFINITY for resource dummy1 on virt-143.
	This will prevent dummy1 from running on virt-143 until the constraint is removed using the 'crm_resource --clear' command or manually with cibadmin
	This will be the case even if virt-143 is the last node in the cluster
	This message can be disabled with --quiet
[root@virt-143 ~]# pcs constraint --full
Location Constraints:
  Resource: dummy1
    Disabled on: virt-143 (score:-INFINITY) (role: Started) (id:cli-ban-dummy1-on-virt-143)

> One of the resource instances WAS moved in both cases so pcs and crm_resource are in sync regarding behavior.


[root@virt-143 ~]# pcs resource ban dummy1-clone
Error: error moving/banning/clearing resource
Resource 'dummy1-clone' not moved: active in 2 locations.
You can prevent 'dummy1-clone' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

[root@virt-143 ~]# 
[root@virt-143 ~]# crm_resource --resource dummy1-clone --ban
Resource 'dummy1-clone' not moved: active in 2 locations.
You can prevent 'dummy1-clone' from running on a specific location with: --ban --node <name>
Error performing operation: Invalid argument

> The difference is in pcs output when targetting DummyBundle which no longer complains about a clone but instead passes the crm_resource message as is. We'll need a more user friendly message in pcs.


The conclusion for 'resource ban' is the same for crm_resource, things are only different for pcs:
 - pcs now doesn't scream about clones anymore but instead pass crm_resource's error message directly, which we might need to modify a little
 - support for banning a bundle in crm_resource is not on par with being able to move a clone, at least in case of just one instance being present due to:
     - 2 active locations being improperly reported 
     - the move constraint of a primitive is incorrectly targeted to a non-existent DummyBundle-0 node

--- Additional comment from Tomas Jelinek on 2023-07-25 15:23:54 CEST ---

Summary after bz1578820 has been resolved:

'pcs resource ban' works as described in comment 0. Such behavior is fine and there is nothing to be fixed. Pcs delegates the banning to crm_resource, including detection of number of location a resource is active in.

'pcs resource move-with constraint' and 'pcs resource move' is needs to be fixed:
* Moving / banning an inner bundle resource from its bundle should be disallowed in pcs. Currently, pcs doesn't check for this and creates a constraint. The constraint has no effect, as expected.
* Moving a 1-replica bundle works, but pcs does not allow it. There is a check in a validator in 'pcs resource move' command which prevents moving all bundle resources. As moving a 1-replica bundle works in pacemaker, an moving multi-replica bundles is prevented by pacemaker, the check in pcs can be removed completely.
* Moving a clone resource works the same as moving a bundle resource and is disallowed by pcs in the same way as well. Removing this limitation from pcs should be considered.