RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1506372 - resource probe return codes other than OCF_NOT_RUNNING all reported as error
Summary: resource probe return codes other than OCF_NOT_RUNNING all reported as error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pacemaker
Version: 8.0
Hardware: All
OS: Linux
low
medium
Target Milestone: pre-dev-freeze
: 8.6
Assignee: Chris Lumens
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 2039982
TreeView+ depends on / blocked
 
Reported: 2017-10-25 19:38 UTC by John Ruemker
Modified: 2022-06-07 17:42 UTC (History)
4 users (show)

Fixed In Version: pacemaker-2.1.2-3.el8
Doc Type: Enhancement
Doc Text:
Feature: If a resource agent returns 2 (parameter invalid locally) or 5 (not installed) for a probe action (non-recurring monitor), Pacemaker will treat that as if the agent returned 7 (not running). Reason: Users sometimes intentionally leave software uninstalled or unconfigured on a node that will never run the associated cluster resource (due to location constraints, etc.). Previously, unless the individual resource agent implemented a workaround, this would result in probe failures showing up in status displays, which would make it more difficult to notice real problems. Result: If a probe returns not installed or not configured, that will no longer be displayed as a failed action, but rather as a reason when showing the resource as stopped.
Clone Of:
: 2039982 (view as bug list)
Environment:
Last Closed: 2022-05-10 14:09:46 UTC
Type: Enhancement
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1514492 0 urgent CLOSED regression in pacemaker-1.1.16-12.el7_4.4.x86_64 / setup with remote-nodes not working anymore 2021-02-22 00:41:40 UTC
Red Hat Issue Tracker CLUSTERQE-5165 0 None None None 2021-10-26 14:02:18 UTC
Red Hat Knowledge Base (Solution) 3304101 0 None None None 2018-07-27 20:01:38 UTC
Red Hat Product Errata RHBA-2022:1885 0 None None None 2022-05-10 14:10:01 UTC

Internal Links: 1514492

Description John Ruemker 2017-10-25 19:38:14 UTC 
Proposal summary: We should consider whether pacemaker's handling and reporting of probe return-codes is ideal, or could be improved - such as by reporting OCF_ERR_INSTALLED and OCF_ERR_CONFIGURED less concerningly, or by having that trigger a ban on nodes that gets reported clearly not as an error but as a resource-state condition.  

Background: We have a customer that had configured their cluster with only some nodes having access to certain storage devices that were to be managed by an LVM resource.  Even though they used a constraint to ban nodes that didn't have access, they were getting probe-errors on those nodes and it was concerning their administrators.  We informed them about resource-discovery and this mostly addresses their initial concern.

However, that sparked a deeper discussion about how resource-agents should indicate in their return codes their discoveries about a node being unable to run that resource, and have pacemaker report it less...aggressively?  scarily?  

pacemaker-explained even says:

  "However, there are two situations where disabling resource discovery is a good idea: when a service is not installed on a node, discovery might return an error (properly written OCF agents will not, so this is usually only seen with other agent types);"
  http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html

However, it seems that pacemaker doesn't actually treat any error-return-codes gracefully on probe (other than of course OCF_NOT_RUNNING).  I've tested 1 - 6 and all general Failed Action reports in 'pcs status' / 'crm_mon' output.  

It seems like there is an opportunity for improvement here, to allow probe returns to influence resource placement, and perhaps have those detected conditions reported in a way that is not so intimidating, and perhaps also removes the need for resource-discovery=no constraints to avoid errors.  

  

Version-Release number of selected component (if applicable): pacemaker-1.1.16-12.el7


How reproducible: Not difficult


Steps to Reproduce: 
1. Configure a resource such as LVM or a custom agent that can be setup with some nodes unprepared to host that resource.

2. Cleanup resource, or just look at initial probe results in 'pcs status'


Actual results: 'pcs status' shows errors  that the administrator may take as problems in the cluster, even though they knew that a resource couldn't run in a particular location. 


Expected results: Agents are able to return from a probe in a specific way that indicates to pacemaker whether that node is eligible to run the resource, and pacemaker reports that condition in a useful way in its outputs. 


Additional info:
Comment 1 John Ruemker 2017-10-26 13:05:01 UTC 
(In reply to John Ruemker from comment #0)
> Proposal summary: We should consider whether pacemaker's handling and
> reporting of probe return-codes is ideal, or could be improved - such as by
> reporting OCF_ERR_INSTALLED and OCF_ERR_CONFIGURED less concerningly, or by
> having that trigger a ban on nodes that gets reported clearly not as an
> error but as a resource-state condition.  
> 

What I was picturing with that suggestion was something like this in crm_mon / pcs status reflecting that we've banned a resource on a node due to its probe result:

 Clone Set: jrummy1-clone [jrummy1]
     Started: [ rhel7-node1.example.com rhel7-node2.example.com rhel7-node3.example.com ]
     Stopped - Not Configured: [ rhel7-node4.example.com ]
     
There could be better terminology to use here, but my point is that we could report this just as a condition of how the resource runs - being banned or stopped on the affected nodes - rather than claiming it as an error.  

Technically its not really an error - the resource wasn't expected to be running and wasn't running - so I feel it shouldn't rise to the level of other errors that get reported in crm_mon output as Failed Actions.  The above example helps make clear what was detected without making it something that has to be acted on.
Comment 2 John Ruemker 2017-10-26 18:40:31 UTC 
To raise one more point that might be useful context here:  The reason I was thinking about this at all to begin with is that we were considering if the LVM agent is returning the correct code when probed on a node without access to the relevant VG.  

This probe result is creating a Failed Action report due to it not returning OCF_NOT_RUNNING.  It is clear there are some agents that avoid this (as is hinted at in the Pacemaker-explained section I referenced earlier), so I wanted to see if LVM should or could being doing a better job with its return.

What I found was that the agents that avoid these errors are ones that mask these "not able to run here" conditions in their probe results.  ocf:heartbeat:named is an example here - if its executing a probe, it translates OCF_ERR_INSTALLED and OCF_ERR_CONFIGURED to OCF_NOT_RUNNING.  While that does successfully avoid the Failed Action report, it does it by masking the real condition that was detected, which isn't exactly ideal.  

When considering what is the best strategy for all of our agents, it feels like it should be preferred to have them report exactly what they detect, and have pacemaker use that information intelligently and only take action or create alerts when it matters.  

And that is my thinking behind this request.
Comment 3 Andrew Beekhof 2017-10-26 23:25:20 UTC 
> While that does successfully avoid the Failed Action report, it does it by masking
> the real condition that was detected, which isn't exactly ideal.  

The problem is that we have no idea if the admin knows the software is not installed there.  That may be why they are looking at pcs status.

If the agent does mask the error, eventually the admin will find out when the start operation fails (unless the admin explicitly tells the cluster not to start it on those nodes).  Granted its not good to find out during an emergency that your one and only backup server didn't have the required bits.

If the agent doesn't mask the error (but the cluster does), then we'll never start it there and not report an error. So the admin may be left wondering why the service isn't running.

If neither the agent nor the cluster masks the error, we'll get bugs like this one from admins annoyed about the noise from persistent expected "failures".


A bit of a no-win situation
Comment 4 Andrew Beekhof 2017-10-27 03:26:59 UTC 
(In reply to John Ruemker from comment #1)
> (In reply to John Ruemker from comment #0)
> > Proposal summary: We should consider whether pacemaker's handling and
> > reporting of probe return-codes is ideal, or could be improved - such as by
> > reporting OCF_ERR_INSTALLED and OCF_ERR_CONFIGURED less concerningly, or by
> > having that trigger a ban on nodes that gets reported clearly not as an
> > error but as a resource-state condition.  
> > 
> 
> What I was picturing with that suggestion was something like this in crm_mon
> / pcs status reflecting that we've banned a resource on a node due to its
> probe result:
> 
>  Clone Set: jrummy1-clone [jrummy1]
>      Started: [ rhel7-node1.example.com rhel7-node2.example.com
> rhel7-node3.example.com ]
>      Stopped - Not Configured: [ rhel7-node4.example.com ]

Oh, I didnt see this before my previous reply but I like it!
Comment 8 Ken Gaillot 2022-01-12 18:34:48 UTC 
Implemented upstream by commits f2e5189~1..ccff6eb

QA: To test, create a ocf:heartbeat:named resource without installing named on at least one node. Before the change, pcs status will show a failed resource action for the probe on any node without named. After, there should be no failed action, but the cluster should not attempt to start named on that node, even if a location constraint prefers it. (Any resource agent without a workaround for probes would work; named just happens to be one.)
Comment 12 Markéta Smazová 2022-02-22 17:20:06 UTC 
before fix
----------

>   [root@virt-320 ~]# rpm -q pacemaker
>   pacemaker-2.1.0-8.el8.x86_64

>   [root@virt-320 ~]# pcs status
>   Cluster name: STSRHTS24683
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-321 (version 2.1.0-8.el8-7c3f660707) - partition with quorum
>     * Last updated: Tue Feb 22 16:41:39 2022
>     * Last change:  Tue Feb 22 16:05:41 2022 by root via cibadmin on virt-320
>     * 2 nodes configured
>     * 2 resource instances configured

>   Node List:
>     * Online: [ virt-320 virt-321 ]

>   Full List of Resources:
>     * fence-virt-320	(stonith:fence_xvm):	 Started virt-320
>     * fence-virt-321	(stonith:fence_xvm):	 Started virt-321

>   Daemon Status:
>     corosync: active/disabled
>     pacemaker: active/disabled
>     pcsd: active/enabled

Install package "bind", which provides "named" on only one of the cluster nodes.
>   [root@virt-321 ~]# rpm -q bind
>   bind-9.11.26-6.el8.x86_64

>   [root@virt-320 ~]# rpm -q bind
>   package bind is not installed

Start "named" on the node, where "bind" is installed:
>   [root@virt-321 ~]# systemctl start named && systemctl is-active named
>   active

Create an ocf:heartbeat:named resource on the node, where "named" is not installed:
>   [root@virt-320 ~]# pcs resource create named1 ocf:heartbeat:named
>   [root@virt-320 ~]# pcs status
>   Cluster name: STSRHTS24683
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-321 (version 2.1.0-8.el8-7c3f660707) - partition with quorum
>     * Last updated: Tue Feb 22 16:47:03 2022
>     * Last change:  Tue Feb 22 16:46:59 2022 by root via cibadmin on virt-320
>     * 2 nodes configured
>     * 3 resource instances configured

>   Node List:
>     * Online: [ virt-320 virt-321 ]

>   Full List of Resources:
>     * fence-virt-320	(stonith:fence_xvm):	 Started virt-320
>     * fence-virt-321	(stonith:fence_xvm):	 Started virt-321
>     * named1	(ocf::heartbeat:named):	 Stopped

>   Failed Resource Actions:
>     * named1_monitor_0 on virt-320 'not installed' (5): call=15, status='complete', exitreason='Setup problem: couldn't find command: /usr/sbin/named', last-rc-change='2022-02-22 16:46:59 +01:00', queued=0ms, exec=35ms

>   Daemon Status:
>     corosync: active/disabled
>     pacemaker: active/disabled
>     pcsd: active/enabled

Pcs status shows a failed resource action for the probe on a node without "named" installed.



after fix
---------

>   [root@virt-036 ~]# rpm -q pacemaker
>   pacemaker-2.1.2-4.el8.x86_64

>   [root@virt-036 ~]# pcs status
>   Cluster name: STSRHTS14728
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-040 (version 2.1.2-4.el8-ada5c3b36e2) - partition with quorum
>     * Last updated: Tue Feb 22 15:37:04 2022
>     * Last change:  Tue Feb 22 11:52:38 2022 by root via cibadmin on virt-036
>     * 2 nodes configured
>     * 2 resource instances configured

>   Node List:
>     * Online: [ virt-036 virt-040 ]

>   Full List of Resources:
>     * fence-virt-036	(stonith:fence_xvm):	 Started virt-036
>     * fence-virt-040	(stonith:fence_xvm):	 Started virt-040

>   Daemon Status:
>     corosync: active/disabled
>     pacemaker: active/disabled
>     pcsd: active/enabled

Install package "bind", which provides "named" on only one of the cluster nodes.
>   [root@virt-040 ~]# rpm -q bind
>   bind-9.11.36-2.el8.x86_64
>   
>   [root@virt-036 ~]# rpm -q bind
>   package bind is not installed

Start "named" on the node, where "bind" is installed:
>   [root@virt-040 ~]# systemctl start named && systemctl is-active named
>   active

Create an ocf:heartbeat:named resource on the node, where "named" is not installed:
>   [root@virt-036 ~]# pcs resource create named2 ocf:heartbeat:named

>   [root@virt-036 ~]# pcs resource
>     * named2	(ocf::heartbeat:named):	 Stopped (not installed) 

>   [root@virt-036 ~]# pcs status
>   Cluster name: STSRHTS14728
>   Cluster Summary:
>     * Stack: corosync
>     * Current DC: virt-040 (version 2.1.2-4.el8-ada5c3b36e2) - partition with quorum
>     * Last updated: Tue Feb 22 15:39:58 2022
>     * Last change:  Tue Feb 22 15:39:50 2022 by root via cibadmin on virt-036
>     * 2 nodes configured
>     * 3 resource instances configured

>   Node List:
>     * Online: [ virt-036 virt-040 ]

>   Full List of Resources:
>     * fence-virt-036	(stonith:fence_xvm):	 Started virt-036
>     * fence-virt-040	(stonith:fence_xvm):	 Started virt-040
>     * named2	(ocf::heartbeat:named):	 Stopped (not installed)

>   Daemon Status:
>     corosync: active/disabled
>     pacemaker: active/disabled
>     pcsd: active/enabled


Pcs status is not showing failed resource action. Resource named2 is stopped.


marking verified in pacemaker-2.1.2-4.el8
Comment 14 errata-xmlrpc 2022-05-10 14:09:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pacemaker bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1885
Note You need to log in before you can comment on or make changes to this bug.