Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
The "pgsql" resource agent and possibly other resources using "crm_mon", during resource stop operations during pacemaker shutdown due to pacemaker reporting "crm_mon: Error: cluster is not available on this node".
For systems running the "pgsql" agent, this leads to a flooding of logs with below errors and delays to services topping:
$ cat node01-Nov11/var/log/messages
---------------------------------------->8--------------------------------------
i-version="2.22" request="crm_mon -1 --output-as=xml">#012 <status code="102"
message="Not connected">#012 <errors>#012 <error>Could not connect to pacemakerd:
Connection refused</error>#012 <error>crm_mon: Connection to cluster failed: Connection
refused</error>#012 </errors>#012 </status>#012</pacemaker-result>".
<28> Nov 10 10:00:53 node01 pgsql(PgsqlRes)[10567]: WARNING: Retrying(remain 68434).
"crm_mon -1 --output-as=xml" failed. rc=102. stdout="<pacemaker-result api-version="2.22"
request="crm_mon -1 --output-as=xml">#012 <status code="102" message="Not connected">#012
<errors>#012 <error>Could not connect to pacemakerd: Connection refused</error>#012
<error>crm_mon: Connection to cluster failed: Connection refused</error>#012
</errors>#012 </status>#012</pacemaker-result>".
---------------------------------------->8--------------------------------------
<28> Nov 11 01:01:56 node01 pgsql(PgsqlRes)[10567]: WARNING: Retrying(remain 16020).
"crm_mon -1 --output-as=xml" failed. rc=102. stdout="<pacemaker-result api-version="2.22"
request="crm_mon -1 --output-as=xml">#012 <status code="102" message="Not connected">#012
<errors>#012 <error>Could not connect to pacemakerd: Connection refused</error>#012
<error>crm_mon: Connection to cluster failed: Connection refused</error>#012
</errors>#012 </status>#012</pacemaker-result>".
<28> Nov 11 01:01:57 node01 pgsql(PgsqlRes)[10567]: WARNING: Retrying(remain 16019).
"crm_mon -1 --output-as=xml" failed. rc=102. stdout="<pacemaker-result api-version="2.22"
request="crm_mon -1 --output-as=xml">#012 <status code="102" message="Not connected">#012
<errors>#012 <error>Could not connect to pacemakerd: Connection refused</error>#012
<error>crm_mon: Connection to cluster failed: Connection refused</error>#012
</errors>#012 </status>#012</pacemaker-result>".
<29> Nov 11 01:01:57 node01 corosync[8452]: [CFG ] Node 5 was shut down by sysadmin
<29> Nov 11 01:01:57 node01 corosync[8452]: [MAIN ] Node was shut down by a signal
<29> Nov 11 01:01:57 node01 corosync[8452]: [SERV ] Unloading all Corosync service engines.
<30> Nov 11 01:01:57 node01 corosync[8452]: [QB ] withdrawing server sockets
<29> Nov 11 01:01:57 node01 corosync[8452]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
<29> Nov 11 01:01:57 node01 corosync[8452]: [CFG ] Node 6 was shut down by sysadmin
<30> Nov 11 01:01:57 node01 corosync[8452]: [QB ] withdrawing server sockets
<29> Nov 11 01:01:57 node01 corosync[8452]: [SERV ] Service engine unloaded: corosync configuration map access
<30> Nov 11 01:01:57 node01 corosync[8452]: [QB ] withdrawing server sockets
---------------------------------------->8--------------------------------------
Version-Release number of selected component (if applicable):
$ grep pacemaker */installed-rpms
pacemaker-2.1.4-5.el8.x86_64 Tue Nov 8 08:36:11 2022
How reproducible:
Reproducible in "pacemaker-2.1.2-4.el8_6.3" and "pacemaker-2.1.4-5.el8_7.2" using steps discussed in below bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1948620#c36
Issue seems consistently reproducible as long as resources agents take long enough to stop.
Steps to Reproduce:
1. Setup a cluster with a resource that takes long enough to stop.
- Dummy resource with "op_sleep" attribute set to 10s used in example.
2. Stop the cluster and run `pcs status` while the resource is stopping.
3. Status shows only `Pacemaker daemons shutting down` and doesn't show resource stopping.
4. Start the cluster, wait for resource `dummy` to start, then stop the cluster again and test XML output.
5. XML output shows error and doesn't show resource stopping.
[root@clusterb-rhel8 ~]# rpm -q pacemaker
pacemaker-2.0.5-9.el8.x86_64
[root@clusterb-rhel8 ~]# pcs cluster stop --all &>/dev/null & sleep 3; crm_mon -1 --output-as=xml | head
[1] 8279
<pacemaker-result api-version="2.3" request="crm_mon -1 --output-as=xml">
<status code="102" message="Not connected">
<errors>
<error>crm_mon: Error: cluster is not available on this node</error>
</errors>
</status>
</pacemaker-result>
Actual results:
Pacemaker reports as not available:
[root@clusterb-rhel8 ~]# pcs cluster stop --all &>/dev/null & sleep 3; crm_mon -1 --output-as=xml | head
[1] 8279
<pacemaker-result api-version="2.3" request="crm_mon -1 --output-as=xml">
<status code="102" message="Not connected">
<errors>
<error>crm_mon: Error: cluster is not available on this node</error>
</errors>
</status>
</pacemaker-result>
Expected results:
Correct crm_mon xml output:
# pcs cluster stop --all &>/dev/null & sleep 3; crm_mon -1 --output-as=xml | head
[2] 19714
<pacemaker-result api-version="2.12" request="crm_mon -1 --output-as=xml">
<summary>
<stack type="corosync"/>
<current_dc present="true" version="2.1.0-8.el8-7c3f660707" name="clustera-rhel8" id="1" with_quorum="true"/>
<last_update time="Wed Nov 30 18:26:23 2022"/>
<last_change time="Wed Nov 30 17:46:33 2022" user="root" client="cibadmin" origin="clusterb-rhel8"/>
<nodes_configured number="2"/>
<resources_configured number="22" disabled="0" blocked="0"/>
<cluster_options stonith-enabled="true" symmetric-cluster="true" no-quorum-policy="stop"
maintenance-mode="false" stop-all-resources="false"/>
</summary>
<nodes>
<node name="clustera-rhel8" id="1" online="true" standby="false" standby_onfail="false" maintenance="false"
pending="false" unclean="false" shutdown="true" expected_up="false" is_dc="true" resources_running="9"
type="member"/>
<node name="clusterb-rhel8" id="2" online="true" standby="false" standby_onfail="false" maintenance="false"
pending="false" unclean="false" shutdown="true" expected_up="false" is_dc="false" resources_running="7"
type="member"/>
</nodes>
Additional info:
This appears to be a regression for below BUG:
1948620 – crm_mon not able to detect resource-status when used in stop-operation of a
Resource Agent during node shutdown
https://bugzilla.redhat.com/show_bug.cgi?id=1948620
A pgsql resource is failed to stop during shutting down in Red Hat Enterprise Linux 8.4 - Red Hat Customer Portal
https://access.redhat.com/solutions/6034771
Description of problem: The "pgsql" resource agent and possibly other resources using "crm_mon", during resource stop operations during pacemaker shutdown due to pacemaker reporting "crm_mon: Error: cluster is not available on this node". For systems running the "pgsql" agent, this leads to a flooding of logs with below errors and delays to services topping: $ cat node01-Nov11/var/log/messages ---------------------------------------->8-------------------------------------- i-version="2.22" request="crm_mon -1 --output-as=xml">#012 <status code="102" message="Not connected">#012 <errors>#012 <error>Could not connect to pacemakerd: Connection refused</error>#012 <error>crm_mon: Connection to cluster failed: Connection refused</error>#012 </errors>#012 </status>#012</pacemaker-result>". <28> Nov 10 10:00:53 node01 pgsql(PgsqlRes)[10567]: WARNING: Retrying(remain 68434). "crm_mon -1 --output-as=xml" failed. rc=102. stdout="<pacemaker-result api-version="2.22" request="crm_mon -1 --output-as=xml">#012 <status code="102" message="Not connected">#012 <errors>#012 <error>Could not connect to pacemakerd: Connection refused</error>#012 <error>crm_mon: Connection to cluster failed: Connection refused</error>#012 </errors>#012 </status>#012</pacemaker-result>". ---------------------------------------->8-------------------------------------- <28> Nov 11 01:01:56 node01 pgsql(PgsqlRes)[10567]: WARNING: Retrying(remain 16020). "crm_mon -1 --output-as=xml" failed. rc=102. stdout="<pacemaker-result api-version="2.22" request="crm_mon -1 --output-as=xml">#012 <status code="102" message="Not connected">#012 <errors>#012 <error>Could not connect to pacemakerd: Connection refused</error>#012 <error>crm_mon: Connection to cluster failed: Connection refused</error>#012 </errors>#012 </status>#012</pacemaker-result>". <28> Nov 11 01:01:57 node01 pgsql(PgsqlRes)[10567]: WARNING: Retrying(remain 16019). "crm_mon -1 --output-as=xml" failed. rc=102. stdout="<pacemaker-result api-version="2.22" request="crm_mon -1 --output-as=xml">#012 <status code="102" message="Not connected">#012 <errors>#012 <error>Could not connect to pacemakerd: Connection refused</error>#012 <error>crm_mon: Connection to cluster failed: Connection refused</error>#012 </errors>#012 </status>#012</pacemaker-result>". <29> Nov 11 01:01:57 node01 corosync[8452]: [CFG ] Node 5 was shut down by sysadmin <29> Nov 11 01:01:57 node01 corosync[8452]: [MAIN ] Node was shut down by a signal <29> Nov 11 01:01:57 node01 corosync[8452]: [SERV ] Unloading all Corosync service engines. <30> Nov 11 01:01:57 node01 corosync[8452]: [QB ] withdrawing server sockets <29> Nov 11 01:01:57 node01 corosync[8452]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0 <29> Nov 11 01:01:57 node01 corosync[8452]: [CFG ] Node 6 was shut down by sysadmin <30> Nov 11 01:01:57 node01 corosync[8452]: [QB ] withdrawing server sockets <29> Nov 11 01:01:57 node01 corosync[8452]: [SERV ] Service engine unloaded: corosync configuration map access <30> Nov 11 01:01:57 node01 corosync[8452]: [QB ] withdrawing server sockets ---------------------------------------->8-------------------------------------- Version-Release number of selected component (if applicable): $ grep pacemaker */installed-rpms pacemaker-2.1.4-5.el8.x86_64 Tue Nov 8 08:36:11 2022 How reproducible: Reproducible in "pacemaker-2.1.2-4.el8_6.3" and "pacemaker-2.1.4-5.el8_7.2" using steps discussed in below bug: https://bugzilla.redhat.com/show_bug.cgi?id=1948620#c36 Issue seems consistently reproducible as long as resources agents take long enough to stop. Steps to Reproduce: 1. Setup a cluster with a resource that takes long enough to stop. - Dummy resource with "op_sleep" attribute set to 10s used in example. 2. Stop the cluster and run `pcs status` while the resource is stopping. 3. Status shows only `Pacemaker daemons shutting down` and doesn't show resource stopping. 4. Start the cluster, wait for resource `dummy` to start, then stop the cluster again and test XML output. 5. XML output shows error and doesn't show resource stopping. [root@clusterb-rhel8 ~]# rpm -q pacemaker pacemaker-2.0.5-9.el8.x86_64 [root@clusterb-rhel8 ~]# pcs cluster stop --all &>/dev/null & sleep 3; crm_mon -1 --output-as=xml | head [1] 8279 <pacemaker-result api-version="2.3" request="crm_mon -1 --output-as=xml"> <status code="102" message="Not connected"> <errors> <error>crm_mon: Error: cluster is not available on this node</error> </errors> </status> </pacemaker-result> Actual results: Pacemaker reports as not available: [root@clusterb-rhel8 ~]# pcs cluster stop --all &>/dev/null & sleep 3; crm_mon -1 --output-as=xml | head [1] 8279 <pacemaker-result api-version="2.3" request="crm_mon -1 --output-as=xml"> <status code="102" message="Not connected"> <errors> <error>crm_mon: Error: cluster is not available on this node</error> </errors> </status> </pacemaker-result> Expected results: Correct crm_mon xml output: # pcs cluster stop --all &>/dev/null & sleep 3; crm_mon -1 --output-as=xml | head [2] 19714 <pacemaker-result api-version="2.12" request="crm_mon -1 --output-as=xml"> <summary> <stack type="corosync"/> <current_dc present="true" version="2.1.0-8.el8-7c3f660707" name="clustera-rhel8" id="1" with_quorum="true"/> <last_update time="Wed Nov 30 18:26:23 2022"/> <last_change time="Wed Nov 30 17:46:33 2022" user="root" client="cibadmin" origin="clusterb-rhel8"/> <nodes_configured number="2"/> <resources_configured number="22" disabled="0" blocked="0"/> <cluster_options stonith-enabled="true" symmetric-cluster="true" no-quorum-policy="stop" maintenance-mode="false" stop-all-resources="false"/> </summary> <nodes> <node name="clustera-rhel8" id="1" online="true" standby="false" standby_onfail="false" maintenance="false" pending="false" unclean="false" shutdown="true" expected_up="false" is_dc="true" resources_running="9" type="member"/> <node name="clusterb-rhel8" id="2" online="true" standby="false" standby_onfail="false" maintenance="false" pending="false" unclean="false" shutdown="true" expected_up="false" is_dc="false" resources_running="7" type="member"/> </nodes> Additional info: This appears to be a regression for below BUG: 1948620 – crm_mon not able to detect resource-status when used in stop-operation of a Resource Agent during node shutdown https://bugzilla.redhat.com/show_bug.cgi?id=1948620 A pgsql resource is failed to stop during shutting down in Red Hat Enterprise Linux 8.4 - Red Hat Customer Portal https://access.redhat.com/solutions/6034771