Bug 1302788 - mandatory constraints set seems no to be honored in more complex scenario
mandatory constraints set seems no to be honored in more complex scenario
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: doc-High_Availability_Add-On_Reference (Show other bugs)
Unspecified Unspecified
unspecified Severity low
: rc
: ---
Assigned To: Steven J. Levine
: Documentation
: 1311749 (view as bug list)
Depends On:
Blocks: 1311749 1376472
  Show dependency treegraph
Reported: 2016-01-28 10:56 EST by michal novacek
Modified: 2017-02-23 13:05 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1311749 (view as bug list)
Last Closed: 2017-02-23 13:05:42 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
pcs cluster cib (48.41 KB, text/plain)
2016-01-28 10:56 EST, michal novacek
no flags Details
pcs cluster report output (1.85 MB, application/x-bzip)
2016-01-28 10:57 EST, michal novacek
no flags Details

  None (edit)
Description michal novacek 2016-01-28 10:56:39 EST
Created attachment 1119188 [details]
pcs cluster cib

Description of problem:

As for my understanding, all colocation constraint set (in my case unordered)
should either run on the same node or not run at all. In the attached cluster
cib there are resource from colocation constraint set running on different

Constraint set is created unordered with require-all=false.

Version-Release number of selected component (if applicable):

How reproducible: always

Steps to Reproduce:
1. check attached cluster cib

Actual results: colocation constraint set resources not collocated

Expected results: colocation constraint set resource running on the same node
or not running at all

Additional info:

[root@duck-01 ~]# pcs -f /tmp/constraints2.xml constraint
Location Constraints:
Ordering Constraints:
Colocation Constraints:
  fs-dist-beaker-www with vg_storage_data (score:INFINITY)
  fs-home with vg_storage_data (score:INFINITY)
  fs-tftpd with vg_storage_data (score:INFINITY)
  export_tftpd with fs-tftpd (score:INFINITY)
  fs-conserver with vg_storage_data (score:INFINITY)
  export_conserver with fs-conserver (score:INFINITY)
  conserver with fs-conserver (score:INFINITY)
  nfs-daemon with vg_storage_data (score:INFINITY)
  Resource Sets:
    set apache nfs-daemon vg_storage_data dhcpd mysql-group vip-zapp sequential=false require-all=false setoptions score=INFINITY
    set beaker-proxy beaker-watchdog beaker-provision beaker-transfer sequential=false require-all=false set apache setoptions score=INFINITY

[root@duck-01 ~]# pcs -f /tmp/constraints2.xml status
Last updated: Thu Jan 28 16:52:09 2016
Last change: Thu Jan 28 16:49:36 2016
Stack: cman
Current DC: duck-02 - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
19 Resources configured

Online: [ duck-01 duck-02 ]

Full list of resources:

 dhcpd  (ocf::heartbeat:Dummy): Started duck-01 
 nfs-daemon     (ocf::heartbeat:Dummy): Started duck-02 
 vg_storage_data        (ocf::heartbeat:Dummy): Started duck-02 
 fs-home        (ocf::heartbeat:Dummy): Started duck-02 
 fs-dist-beaker-www     (ocf::heartbeat:Dummy): Started duck-02 
 fs-conserver   (ocf::heartbeat:Dummy): Started duck-02 
 fs-tftpd       (ocf::heartbeat:Dummy): Started duck-02 
 conserver      (ocf::heartbeat:Dummy): Started duck-02 
 export_conserver       (ocf::heartbeat:Dummy): Started duck-02 
 export_tftpd   (ocf::heartbeat:Dummy): Started duck-02 
 Resource Group: mysql-group
     vg_shared_db       (ocf::heartbeat:Dummy): Started duck-01 
     fs-dist-db (ocf::heartbeat:Dummy): Started duck-01 
     mysql      (ocf::heartbeat:Dummy): Started duck-01 
 vip-zapp       (ocf::heartbeat:Dummy): Started duck-01 
 apache (ocf::heartbeat:Dummy): Started duck-01 
 beaker-proxy   (ocf::heartbeat:Dummy): Started duck-01 
 beaker-watchdog        (ocf::heartbeat:Dummy): Started duck-01 
 beaker-provision       (ocf::heartbeat:Dummy): Started duck-01 
 beaker-transfer        (ocf::heartbeat:Dummy): Started duck-01
Comment 1 michal novacek 2016-01-28 10:57 EST
Created attachment 1119189 [details]
pcs cluster report output
Comment 3 Ken Gaillot 2016-02-04 16:07:04 EST
The syntax and documentation for colocation sets is quite confusing (even to me). I will work on the documentation, but to clarify for this case, this is my understanding:

* require-all=false is meaningful only for ordering sets, not colocation sets, so it is irrelevant here

* a colocation set with sequential=false, counterintuitively, does not add any colocation dependency between the items in that set. Instead, it allows the set to be in a colocation constraint with other sets, without its members being affected by each other. Therefore, it only makes sense if there is another set in the constraint.

A constraint like your first one, of the form:

  colocation set A B C sequential=false

is a null statement, with no effect.

A constraint like your second one, of the form:

  colocation set A B C sequential=false set D

means that A, B, and C must be colocated with D, but A, B, and C do not have any explicit relationship with each other (so any of them can fail without affecting the others).

Also, there is always some confusion between ordering constraints and colocation constraints. Colocation of "A with B" (including the colocation set equivalent) means that B will be assigned to a node first, but that does not mean that B will necessarily be started first. Without a separate ordering constraint, A and B may be started simultaneously (or even in reverse order) -- it just will be guaranteed to be on the same node.

I'll clone this BZ for pcs so we can have it reject require-all=false for colocation sets, and reject colocation sets with sequential=false unless there is another set in the constraint.
Comment 4 michal novacek 2016-02-08 09:22:31 EST
Thanks Ken, that make sense. Something like you write here should go to the documentation to avoid future confusion.
Comment 5 Ken Gaillot 2016-02-08 11:41:20 EST
To elaborate further, if the goal is to place one resource with another if possible, but allow it to run even if the other resource isn't running, then a colocation constraint or set with a score less than INFINITY is sufficient. For example:

  pcs constraint colocation add B with A 1000

would place B with A if A is running, but allow B to run anywhere if A is not running. And:

  pcs constraint colocation set A B C setoptions score=1000

would place A first, then place B with A if possible (otherwise anywhere), then place C with B if possible (otherwise anywhere).
Comment 6 Ken Gaillot 2016-02-24 16:33:17 EST
Reassigning to RHEL 7.2 docs. Same concerns apply to RHEL 6.

Upstream documentation has been updated: http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html-single/Pacemaker_Explained/index.html#ch-constraints

The relevant upstream changes are: https://github.com/ClusterLabs/pacemaker/pull/927/files

The key items for RH docs:

* In 6.2.3. Ordered Resource Sets, role is not an option for these sets.

* In 6.3.3 Colocating Sets of Resources, action and require-all are not options for these sets.

* In both those sections, we should make it clearer that sequential=false only makes sense if multiple sets are listed in the constraint, otherwise the constraint has no effect.

* The description of require-all in ordered sets could be more detailed; upstream documentation should be a good starting point.
Comment 8 Steven J. Levine 2016-09-15 10:01:14 EDT
*** Bug 1311749 has been marked as a duplicate of this bug. ***
Comment 9 Steven J. Levine 2016-09-15 15:12:42 EDT

I have made a few updates to the RHEL 7.3 version of the Pacemaker doc but this might not be quite enough explanation, I'm not sure.

The latest version of the doc is here:


Ordered Resource Sets is Section 7.2.3

Colocating Sets of Resources is Section 7.3.3

Comment 10 Ken Gaillot 2016-09-16 14:38:36 EDT
7.2.3 Ordered Resource Sets:

It is probably worth mentioning that if the goal is to make a set of resources both colocated together and started in order, a resource group (section 6.5) is usually the best choice. Ordered sets allow the resources to be ordered, without necessarily being colocated on the same node.

I'm not sure if it needs to be changed, but where you say "following the options parameter", there is no options parameter explicitly specified, rather the options are simply listed after the set members. For example "pcs constraint order set A B sequential=false" (not "... options sequential=false"). (By contrast, "setoptions" is explicitly needed.)

The description for "sequential" for ordered sets should be along the lines of:

"sequential, which can be set to true or false to indicate whether the members of the set must be ordered relative to each other.

Setting sequential to false allows a set to be ordered relative to other sets in the ordering constraint, without its members being ordered relative to each other. Therefore, this option makes sense only if multiple sets are listed in the constraint; otherwise, the constraint has no effect."

The key distinction is that an ordered set is always ordered relative to other sets in the constraint, but sequential controls whether the members of the set are ordered relative to each other.

7.3.3 Colocating Sets of Resources:

Probably a similar reference to groups is useful here. Colocation sets allow the resources to be placed on the same node, without requiring them to be started in order.

The same comment for ordering applies about the "options parameter".

The description of "sequential" for colocated sets should be along the lines of:

"sequential, which can be set to true or false to indicate whether the members of the set must be colocated with each other.

Setting sequential to false allows the members of this set to be colocated with another set listed later in the constraint, regardless of which members of this set are active. Therefore, this option makes sense only if another set is listed after this one in the constraint; otherwise, the constraint has no effect."

One point of possible confusion that may be worth pointing out is that when listing members of a set, each member is colocated with the one *before* it. For example, "set A B" means "B is colocated with A". However, when listing multiple sets, each set is colocated with the one *after* it. For example, "set C D sequential=false set A B" means "set C D (where C and D have no relation between each other) is colocated with set A B (where B is colocated with A)".
Comment 11 Steven J. Levine 2016-09-16 17:51:59 EDT

I have made the updates in comment 10.  I wasn't sure if your point of possible confusion was information you wanted me to rephrase or use verbatim but it seemed clear just as is so I left it as is.

If you want to check the updates they are here:

Comment 12 Steven J. Levine 2016-10-21 14:45:19 EDT
No further updates from Ken and 7.3 is going out the door so while I'm pretty sure this is finished I'm moving it to 7.4 for now just to be sure Ken is ok with what we have before closing this.

Note You need to log in before you can comment on or make changes to this bug.