Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1650883[details]
pcs cluster report output starting with the time of the reboot
Description of problem:
After restarting the node, "pcs status" sometimes hangs for many minutes, and sometimes returns with this error output:
====
[root@porteri:~]# pcs status
Cluster name: porteri
WARNINGS:
Critical: Unable to get stonith-history
Connection to the cluster-daemons terminated
Reading stonith-history failed
Daemon Status:
corosync: active/disabled
pacemaker: active/enabled
pcsd: inactive/disabled
====
Adding -V to the pacemaker ExecStart command, I see other error messages logged, including:
====
Jan 8 13:46:25 porteri stonith-ng[87063]: error: Connection to cib_rw failed
Jan 8 13:46:25 porteri stonith-ng[87063]: error: Connection to cib_rw[0x55555579ff10] closed (I/O condition=1)
Jan 8 13:46:25 porteri stonith-ng[87063]: notice: Connection to the CIB terminated. Shutting down.
Jan 8 13:46:25 porteri lrmd[87064]: error: Connection to stonith-ng failed
Jan 8 13:46:25 porteri lrmd[87064]: error: Connection to stonith-ng[0x55555578ec00] closed (I/O condition=17)
Jan 8 13:46:25 porteri pacemakerd[87061]: notice: Respawning failed child process: stonith-ng
Jan 8 13:46:25 porteri stonith-ng[87659]: notice: Additional logging available in /var/log/pacemaker-direct.log
Jan 8 13:46:25 porteri stonith-ng[87659]: notice: Connecting to cluster infrastructure: corosync
Jan 8 13:46:25 porteri stonith-ng[87659]: notice: Node porteri state is now member
Jan 8 13:46:25 porteri crmd[87067]: error: Connection to stonith-ng failed
Jan 8 13:46:25 porteri crmd[87067]: error: Connection to stonith-ng[0x555555aa9570] closed (I/O condition=17)
====
Stopping and restarting pacemaker does not change the symptoms.
Pacemaker otherwise seems to function - I stopped a resource on one node and pacemaker restarted it properly, I put a node into standby and the resources were moved properly, and I unstandby'd the node and the resources moved back.
I consistently saw the symptoms above after restarted the pacemaker service while troubleshooting. When I went back later, perhaps 30 minutes or less, pcs status returned quickly and with the actual cluster state. So it's possible this is made more likely when pacemaker is updating internal state frequently.
Version-Release number of selected component (if applicable):
pacemaker-1.1.20-5.el7_7.2.x86_64
How reproducible:
Unknown. I saw this after a node reboot, I have only seen these symptoms on one cluster. I did not see them on my test cluster.
Steps to Reproduce:
1. It may be that this is a consequence of updating from pacemaker-1.1.20-5.el7_7.1.x86_64 to pacemaker-1.1.20-5.el7_7.2.x86_64 via yum update.
Actual results:
Expected results:
pcs status immediately returns the resources and their states if systemctl status pacemaker indicates active.
Additional info:
On the machine where I see the problem, I ran the following (slightly abbreviated):
pcs cluster kill; systemctl start pacemaker; for x in $(seq 100); do time pcs status; done
I did this a few times, and find there is a pattern to the time it takes for pcs status to return.
1. Most resources are stopped: About 0.7 seconds
2. During transition where many resources are going from stopped => started: 30 seconds to several minutes
3. After all resources are started: About 1.4 seconds
For details, see pcs-status-time-test.4.txt (I will attach)
This is the same issue as in Bug 1625671, which is scheduled for 7.8. However that one also depends on Bug 1710988 which is not scheduled for a release yet. Feel free to comment on either or both of those to show it's being hit in the wild. If you are interested in z-streams and/or hot fixes, please contact support to get that process started.
*** This bug has been marked as a duplicate of bug 1625671 ***
Created attachment 1650883 [details] pcs cluster report output starting with the time of the reboot Description of problem: After restarting the node, "pcs status" sometimes hangs for many minutes, and sometimes returns with this error output: ==== [root@porteri:~]# pcs status Cluster name: porteri WARNINGS: Critical: Unable to get stonith-history Connection to the cluster-daemons terminated Reading stonith-history failed Daemon Status: corosync: active/disabled pacemaker: active/enabled pcsd: inactive/disabled ==== Adding -V to the pacemaker ExecStart command, I see other error messages logged, including: ==== Jan 8 13:46:25 porteri stonith-ng[87063]: error: Connection to cib_rw failed Jan 8 13:46:25 porteri stonith-ng[87063]: error: Connection to cib_rw[0x55555579ff10] closed (I/O condition=1) Jan 8 13:46:25 porteri stonith-ng[87063]: notice: Connection to the CIB terminated. Shutting down. Jan 8 13:46:25 porteri lrmd[87064]: error: Connection to stonith-ng failed Jan 8 13:46:25 porteri lrmd[87064]: error: Connection to stonith-ng[0x55555578ec00] closed (I/O condition=17) Jan 8 13:46:25 porteri pacemakerd[87061]: notice: Respawning failed child process: stonith-ng Jan 8 13:46:25 porteri stonith-ng[87659]: notice: Additional logging available in /var/log/pacemaker-direct.log Jan 8 13:46:25 porteri stonith-ng[87659]: notice: Connecting to cluster infrastructure: corosync Jan 8 13:46:25 porteri stonith-ng[87659]: notice: Node porteri state is now member Jan 8 13:46:25 porteri crmd[87067]: error: Connection to stonith-ng failed Jan 8 13:46:25 porteri crmd[87067]: error: Connection to stonith-ng[0x555555aa9570] closed (I/O condition=17) ==== Stopping and restarting pacemaker does not change the symptoms. Pacemaker otherwise seems to function - I stopped a resource on one node and pacemaker restarted it properly, I put a node into standby and the resources were moved properly, and I unstandby'd the node and the resources moved back. I consistently saw the symptoms above after restarted the pacemaker service while troubleshooting. When I went back later, perhaps 30 minutes or less, pcs status returned quickly and with the actual cluster state. So it's possible this is made more likely when pacemaker is updating internal state frequently. Version-Release number of selected component (if applicable): pacemaker-1.1.20-5.el7_7.2.x86_64 How reproducible: Unknown. I saw this after a node reboot, I have only seen these symptoms on one cluster. I did not see them on my test cluster. Steps to Reproduce: 1. It may be that this is a consequence of updating from pacemaker-1.1.20-5.el7_7.1.x86_64 to pacemaker-1.1.20-5.el7_7.2.x86_64 via yum update. Actual results: Expected results: pcs status immediately returns the resources and their states if systemctl status pacemaker indicates active. Additional info: