Thanks for the report. A few issues are entangled here:

* Rules may be used in multiple contexts (cluster options, resource defaults, location constraints, etc.). Certain rule configuration elements are only used in certain contexts, but Pacemaker's CIB schema does not prevent users from configuring those in other contexts, where they will be ignored or always make the rule evaluation false.

* Upstream documentation is silent on whether node attribute expressions are allowed in resource defaults. It does mention that they are allowed in an individual resource's configuration, and that they are disallowed in cluster-wide options. I don't think we have much downstream documentation on the subject, but instead rely on pcs's man page.

* Node attribute expressions are currently always treated as false in resource defaults, but pcs allows it, and there's a reasonable argument Pacemaker should support them. I would consider this an RFE.

The fix for the first issue is straightforward, but unfortunately we can't do it right now because it could potentially invalidate existing configurations that currently work fine. The ideal time to do it would be the next time upstream does a major-version schema update in Pacemaker, which would include an XSL transform to drop unsupported rules from existing configurations. That will likely be in the RHEL 10 time frame. In the meantime, we could add a configuration warning when a rule is used in an unsupported context.

The second issue is also straightforward, and we can take care of it once we decide what to do about the third issue. My first thought is that it should be reasonably simple to implement, but there may be some reason I'm overlooking why it was never hooked up to begin with.

To sum up, for this BZ, we can:
1. Update the upstream documentation to clearly indicate where each element of rule syntax can be used.
2. Start logging a configuration warning when rule elements are used in unsupported contexts.
3. Investigate whether supporting node attribute expressions in resource defaults is feasible.

(In reply to Ken Gaillot from comment #1)
> * Node attribute expressions are currently always treated as false in
> resource defaults, but pcs allows it, and there's a reasonable argument
> Pacemaker should support them. I would consider this an RFE.

Well, sorry about that. :) I was working with Ken's comment in bz1869399 comment0, "Also, the help for "pcs resource op defaults set create" and "pcs resource defaults set create" implies that only resource and operation expressions are allowed there. Date/time expressions and node attribute expressions are also allowed.". Based on that, it was my impression node attribute expressions are actually supported by Pacemaker. That's why I went ahead and added support for them in pcs.

If node attribute expressions are not going to be supported by pacemaker, feel free to move this bz to pcs. We can modify rule validators in pcs so that they prevent node attribute expressions being used.

(In reply to Tomas Jelinek from comment #2)
> (In reply to Ken Gaillot from comment #1)
> > * Node attribute expressions are currently always treated as false in
> > resource defaults, but pcs allows it, and there's a reasonable argument
> > Pacemaker should support them. I would consider this an RFE.
> 
> Well, sorry about that. :) I was working with Ken's comment in bz1869399

Don't listen to that guy, he doesn't know what he's talking about :-P

> comment0, "Also, the help for "pcs resource op defaults set create" and "pcs
> resource defaults set create" implies that only resource and operation
> expressions are allowed there. Date/time expressions and node attribute
> expressions are also allowed.". Based on that, it was my impression node

Technically node attribute expressions are allowed, they're just always false ;-)

Obviously it's easy to think they would be supported, even having some familiarity with the code, so I'm leaning to adding the support in Pacemaker if there's not some large design roadblock. Since there's no documentation, I'll still consider it an RFE.

> attribute expressions are actually supported by Pacemaker. That's why I went
> ahead and added support for them in pcs.
> 
> If node attribute expressions are not going to be supported by pacemaker,
> feel free to move this bz to pcs. We can modify rule validators in pcs so
> that they prevent node attribute expressions being used.

I'll target 8.5 for the pacemaker feature since QA is at capacity for 8.4. If QA winds up having more time near the end, we might be able to get this in for 8.4. It's up to you what to do about pcs in 8.4.

After further discussion, we've decided that this bz should focus on the narrow issue of node attribute expressions being allowed in resource defaults. We have a good chance of getting that in 8.4.

Could you attach the CIB after adding the resources in step #3, but before moving the one resource?  I'd like to verify that I am writing a test case for the correct thing here.  Thanks.

After further investigation, we determined that it is correct but undocumented behavior that node attribute expressions are not supported for *meta-attributes*, except in the context of an operation or operation defaults. Node attribute expressions do not work with meta-attributes in a resource's configuration, resource defaults, or cluster properties.

Node attribute expressions should work for resource options (instance attributes), whether in the resource configuration or resource defaults, so it is correct that pcs allows node attribute expressions in resource defaults. (That can be used in testing the original bz.) It would probably be a good idea for pcs to reject the combination of meta and node attribute rules for resource defaults.

This makes sense when you consider that only an operation is directly associated with a single node. Most meta-attributes would make no sense being node-specific (target-role, allow-migrate, etc.). While some might (resource-stickiness, migration-threshold), that would be an unusual use case, and I'd rather wait until there is customer demand before formalizing that as an RFE.

I'm not sure whether this needs to be documented in RHEL. I believe rules are currently documented only in the context of location constraints (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_determining-resource-location-with-rules-configuring-and-managing-high-availability-clusters) so there's no existing place for it. Possibly the pcs man page could be updated for "pcs resource defaults".

Pacemaker should enforce proper rule syntax usage more strictly via its XML schema, but for compatibility reasons that is not feasible in the RHEL 8 time frame.