Bug 2133546

Summary: pgsql agent fails at cluster shutdown due to crm_mon regression
Product: Red Hat Enterprise Linux 9 Reporter: Ken Gaillot <kgaillot>
Component: pacemakerAssignee: Reid Wahl <nwahl>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: urgent Docs Contact: Steven J. Levine <slevine>
Priority: urgent    
Version: 9.1CC: cfeist, cluster-maint, cluster-qe, jrehova, msmazova, nwahl, sbradley, slevine
Target Milestone: rcKeywords: Regression, Triaged, ZStream
Target Release: 9.2Flags: pm-rhel: mirror+
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: pacemaker-2.1.5-1.el9 Doc Type: Bug Fix
Doc Text:
.Cluster resources that call `crm_mon` now stop cleanly at shutdown Previously, the `crm_mon` utility returned a nonzero exit status while Pacemaker was in the process of shutting down. Resource agents that called `crm_mon` in their monitor action, such as `ocf:heartbeat:pqsql`, could incorrectly return a failure at cluster shutdown. With this fix, `crm_mon` returns success even if the cluster is in the process of shutting down. Resources that call `crm_mon` now stop cleanly at cluster shutdown.
Story Points: ---
Clone Of: 2133497
: 2133911 (view as bug list) Environment:
Last Closed: 2023-05-09 07:18:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 2.1.5
Embargoed:
Bug Depends On: 2133497    
Bug Blocks: 2133911    

Description Ken Gaillot 2022-10-10 20:01:38 UTC
+++ This bug was initially created as a clone of Bug #2133497 +++

Description of problem: If the ocf:heartbeat:pgsql resource agent runs a monitor while Pacemaker is in the process of shutting down, it will incorrectly get an error.


Version-Release number of selected component (if applicable): RHEL 9.1 packages


How reproducible: reliably


Steps to Reproduce:
1. Configure a cluster with an ocf:heartbeat:pgsql resource with a frequent monitor. It helps to configure a slow resource (e.g. Dummy with a stop delay) to make the problem more likely.
2. Shut down the cluster.

Actual results: The pgsql resource fails a monitor.


Expected results: The cluster shuts down cleanly.


Additional info: This is a regression in the 9.1 packages.

Comment 5 Ken Gaillot 2022-10-12 21:42:35 UTC
Fixed in upstream main branch as of commit 2d4a36c0f

Comment 10 jrehova 2022-12-05 16:53:49 UTC
* 2 nodes cluster

Version of pacemaker:

> [root@virt-261 ~]# rpm -q pacemaker
> pacemaker-2.1.5-2.el9.x86_64

Created a resource with a stop delay and waited until it started:

> [root@virt-261 ~]# pcs resource create dummy ocf:pacemaker:Dummy op_sleep=10

> [root@virt-261 ~]# pcs resource
>   * dummy	(ocf:pacemaker:Dummy):	 Started virt-261

Stoped the cluster and run "pcs status" while the resource was stopping:

> [root@virt-261 ~]# pcs cluster stop --all &>/dev/null & sleep 10; pcs status; echo $?
> [1] 345834
> Cluster name: STSRHTS28390
> Status of pacemakerd: 'Pacemaker daemons are shutting down' (last updated 2022-12-05 17:40:11 +01:00)
> Cluster Summary:
>   * Stack: corosync
>   * Current DC: virt-261 (version 2.1.5-2.el9-631339ca5aa) - MIXED-VERSION partition with quorum
>   * Last updated: Mon Dec  5 17:40:11 2022
>   * Last change:  Mon Dec  5 17:35:31 2022 by root via cibadmin on virt-261
>   * 2 nodes configured
>   * 3 resource instances configured
> 
> Node List:
>   * Node virt-262: pending
>   * Online: [ virt-261 ]
> 
> Full List of Resources:
>   * fence-virt-261	(stonith:fence_xvm):	 Stopped
>   * fence-virt-262	(stonith:fence_xvm):	 Stopped
>   * dummy	(ocf:pacemaker:Dummy):	 Stopping virt-261
> 
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: inactive/disabled
>   pcsd: active/enabled
> 0

Result: The cluster shuted down cleanly, crm_mon doesn't fail.

Comment 13 errata-xmlrpc 2023-05-09 07:18:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pacemaker bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2150