Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2042550

Summary:	pacemaker.service fails to start if enabling corosync CFG shutdown tracker returns CS_ERR_TRY_AGAIN
Product:	Red Hat Enterprise Linux 8	Reporter:	Ken Gaillot <kgaillot>
Component:	pacemaker	Assignee:	Ken Gaillot <kgaillot>
Status:	CLOSED ERRATA	QA Contact:	cluster-qe <cluster-qe>
Severity:	medium	Docs Contact:
Priority:	high
Version:	8.6	CC:	cluster-maint, cluster-qe, msmazova, nwahl, phagara, sbradley
Target Milestone:	rc	Keywords:	Triaged
Target Release:	8.6	Flags:	pm-rhel: mirror+
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	pacemaker-2.1.2-3.el8	Doc Type:	Bug Fix
Doc Text:	Cause: In some network configurations, Pacemaker can fail to register with Corosync's shutdown tracker on the first attempt. Consequence: Pacemaker exits and is respawned repeatedly until systemd gives up, and manual intervention is required to return the node to the cluster. Fix: Pacemaker now retries the connection if the first attempt fails. Result: The node is able to rejoin the cluster automatically.	Story Points:	---
Clone Of:	2042367	Environment:
Last Closed:	2022-05-10 14:10:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2042367
Bug Blocks:

Description Ken Gaillot 2022-01-19 17:08:26 UTC

+++ This bug was initially created as a clone of Bug #2042367 +++

Description of problem:

Under some circumstances (see below), corosync is returning CS_ERR_TRY_AGAIN (for up to 30s here, maybe more in other situations?) to pacemaker's requests to enable shutdown tracker. This causes pacemaker to immediately (~0.1s) exit with a critical error.

Systemd keeps trying to start pacemaker.service again and again with only RestartSec=100ms delay between retries, until the service restart burst rate-limit (`DefaultStartLimitIntervalSec` & `DefaultStartLimitBurst`) is hit (which takes only around 5*(0.1s+0.1s)=1 second with default systemd config), then marks the service as failed.

Afterwards, manual intervention (starting pacemaker.service) is required to restore the node back into working condition.

A patch adding a retry loop around shutdown tracker enable request has been created already, but not tested yet: https://github.com/ClusterLabs/pacemaker/pull/2611


Version-Release number of selected component (if applicable):
pacemaker-2.1.2-2.el9.x86_64
corosync-3.1.5-3.el9.x86_64


How reproducible:
reliable, likely dependent on network setup/topology (I suspect STP timers on the virtual bridge in this case)


Steps to Reproduce:
1. configure a 2-node cluster with corosync-qdevice in either lms or ffsplit mode with lowest-corosync-node-id tie-breaker (default)
2. enable pacemaker to start on boot
3. kill (eg. `halt -fin`) the node with the higher corosync node id
4. wait for it to get fenced and boot again
5. pacemaker service hits systemd burst restart rate-limit


Actual results:

Pacemaker fails to enable corosync cfg shutdown tracker and immediately exits with a critical error. Systemd immediately attempts to restart the failed pacemaker.service, which immediately fails again. This repeats until the systemd service restart rate-limit is hit, after which pacemaker.service remains in failed state and manual intervention is required before the node is able to join cluster again.

>     [root@virt-023 ~]# journalctl -u pacemaker.service -u corosync.service -o short-monotonic
>     -- Journal begins at Tue 2022-01-18 12:27:54 CET, ends at Tue 2022-01-18 16:30:54 CET. --
>     [    4.285754] virt-023 systemd[1]: Starting Corosync Cluster Engine...
>     [    4.325882] virt-023 corosync[773]:   [MAIN  ] Corosync Cluster Engine 3.1.5 starting up
>     [    4.326957] virt-023 corosync[773]:   [MAIN  ] Corosync built-in features: dbus systemd xmlconf vqsim nozzle snmp pie relro bindnow
>     [    4.379839] virt-023 corosync[773]:   [TOTEM ] Initializing transport (Kronosnet).
>     [    4.751284] virt-023 corosync[773]:   [TOTEM ] totemknet initialized
>     [    4.751349] virt-023 corosync[773]:   [KNET  ] common: crypto_nss.so has been loaded from /usr/lib64/kronosnet/crypto_nss.so
>     [    4.852202] virt-023 corosync[773]:   [SERV  ] Service engine loaded: corosync configuration map access [0]
>     [    4.852400] virt-023 corosync[773]:   [QB    ] server name: cmap
>     [    4.852420] virt-023 corosync[773]:   [SERV  ] Service engine loaded: corosync configuration service [1]
>     [    4.852455] virt-023 corosync[773]:   [QB    ] server name: cfg
>     [    4.852482] virt-023 corosync[773]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
>     [    4.852513] virt-023 corosync[773]:   [QB    ] server name: cpg
>     [    4.852529] virt-023 corosync[773]:   [SERV  ] Service engine loaded: corosync profile loading service [4]
>     [    4.852543] virt-023 corosync[773]:   [QUORUM] Using quorum provider corosync_votequorum
>     [    4.852650] virt-023 corosync[773]:   [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
>     [    4.852727] virt-023 corosync[773]:   [QB    ] server name: votequorum
>     [    4.852748] virt-023 corosync[773]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
>     [    4.852762] virt-023 corosync[773]:   [QB    ] server name: quorum
>     [    4.852842] virt-023 corosync[773]:   [TOTEM ] Configuring link 0
>     [    4.852867] virt-023 corosync[773]:   [TOTEM ] Configured link number 0: local addr: 2620:52:0:25a4:1800:ff:fe00:17, port=5405
>     [    4.858084] virt-023 corosync[773]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
>     [    4.858190] virt-023 corosync[773]:   [KNET  ] host: host: 1 has no active links
>     [    4.861527] virt-023 corosync[773]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
>     [    4.861900] virt-023 corosync[773]:   [KNET  ] host: host: 1 has no active links
>     [    4.861947] virt-023 corosync[773]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
>     [    4.861986] virt-023 corosync[773]:   [KNET  ] host: host: 1 has no active links
>     [    4.862003] virt-023 corosync[773]:   [QUORUM] Sync members[1]: 2
>     [    4.862032] virt-023 corosync[773]:   [QUORUM] Sync joined[1]: 2
>     [    4.862048] virt-023 corosync[773]:   [TOTEM ] A new membership (2.4d) was formed. Members joined: 2
>     [    4.862064] virt-023 corosync[773]:   [QUORUM] Members[1]: 2
>     [    4.862080] virt-023 corosync[773]:   [MAIN  ] Completed service synchronization, ready to provide service.
>     [    4.864038] virt-023 systemd[1]: Started Corosync Cluster Engine.
>     [    8.916922] virt-023 corosync[773]:   [KNET  ] rx: host: 1 link: 0 is up
>     [    8.917239] virt-023 corosync[773]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
>     [    9.031633] virt-023 corosync[773]:   [QUORUM] Sync members[2]: 1 2
>     [    9.031787] virt-023 corosync[773]:   [QUORUM] Sync joined[1]: 1
>     [    9.031804] virt-023 corosync[773]:   [VOTEQ ] waiting for quorum device Qdevice poll (but maximum for 30000 ms)
>     [    9.031820] virt-023 corosync[773]:   [TOTEM ] A new membership (1.51) was formed. Members joined: 1
>     [    9.110793] virt-023 corosync[773]:   [KNET  ] pmtud: Global data MTU changed to: 1157
>     [    9.134123] virt-023 systemd[1]: Started Pacemaker High Availability Cluster Manager.
>     [    9.151630] virt-023 pacemakerd[1947]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
>     [    9.213969] virt-023 pacemakerd[1947]:  notice: Starting Pacemaker 2.1.2-2.el9
>     [    9.259625] virt-023 pacemakerd[1947]:  crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN
>     [    9.261144] virt-023 systemd[1]: pacemaker.service: Main process exited, code=exited, status=76/PROTOCOL
>     [    9.261653] virt-023 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
>     [    9.612498] virt-023 systemd[1]: pacemaker.service: Scheduled restart job, restart counter is at 1.
>     [    9.612959] virt-023 systemd[1]: Stopped Pacemaker High Availability Cluster Manager.
>     [    9.615763] virt-023 systemd[1]: Started Pacemaker High Availability Cluster Manager.
>     [    9.622434] virt-023 pacemakerd[1954]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
>     [    9.628555] virt-023 pacemakerd[1954]:  notice: Starting Pacemaker 2.1.2-2.el9
>     [    9.630961] virt-023 pacemakerd[1954]:  crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN
>     [    9.632937] virt-023 systemd[1]: pacemaker.service: Main process exited, code=exited, status=76/PROTOCOL
>     [    9.632986] virt-023 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
>     [    9.984776] virt-023 systemd[1]: pacemaker.service: Scheduled restart job, restart counter is at 2.
>     [    9.984993] virt-023 systemd[1]: Stopped Pacemaker High Availability Cluster Manager.
>     [    9.987023] virt-023 systemd[1]: Started Pacemaker High Availability Cluster Manager.
>     [    9.994160] virt-023 pacemakerd[1990]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
>     [   10.002388] virt-023 pacemakerd[1990]:  notice: Starting Pacemaker 2.1.2-2.el9
>     [   10.004448] virt-023 pacemakerd[1990]:  crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN
>     [   10.006381] virt-023 systemd[1]: pacemaker.service: Main process exited, code=exited, status=76/PROTOCOL
>     [   10.006435] virt-023 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
>     [   10.359510] virt-023 systemd[1]: pacemaker.service: Scheduled restart job, restart counter is at 3.
>     [   10.359833] virt-023 systemd[1]: Stopped Pacemaker High Availability Cluster Manager.
>     [   10.360353] virt-023 systemd[1]: Started Pacemaker High Availability Cluster Manager.
>     [   10.369133] virt-023 pacemakerd[2002]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
>     [   10.381569] virt-023 pacemakerd[2002]:  notice: Starting Pacemaker 2.1.2-2.el9
>     [   10.381827] virt-023 pacemakerd[2002]:  crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN
>     [   10.383655] virt-023 systemd[1]: pacemaker.service: Main process exited, code=exited, status=76/PROTOCOL
>     [   10.383695] virt-023 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
>     [   10.734612] virt-023 systemd[1]: pacemaker.service: Scheduled restart job, restart counter is at 4.
>     [   10.734750] virt-023 systemd[1]: Stopped Pacemaker High Availability Cluster Manager.
>     [   10.737113] virt-023 systemd[1]: Started Pacemaker High Availability Cluster Manager.
>     [   10.744137] virt-023 pacemakerd[2004]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
>     [   10.753075] virt-023 pacemakerd[2004]:  notice: Starting Pacemaker 2.1.2-2.el9
>     [   10.755228] virt-023 pacemakerd[2004]:  crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN
>     [   10.757386] virt-023 systemd[1]: pacemaker.service: Main process exited, code=exited, status=76/PROTOCOL
>     [   10.757444] virt-023 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
>     [   11.108323] virt-023 systemd[1]: pacemaker.service: Scheduled restart job, restart counter is at 5.
>     [   11.108772] virt-023 systemd[1]: Stopped Pacemaker High Availability Cluster Manager.
>     [   11.108808] virt-023 systemd[1]: pacemaker.service: Start request repeated too quickly.
>     [   11.108828] virt-023 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
>     [   11.108848] virt-023 systemd[1]: Failed to start Pacemaker High Availability Cluster Manager.
>     [   19.735837] virt-023 corosync[773]:   [QUORUM] This node is within the primary component and will provide service.
>     [   19.737102] virt-023 corosync[773]:   [QUORUM] Members[2]: 1 2
>     [   19.737134] virt-023 corosync[773]:   [MAIN  ] Completed service synchronization, ready to provide service.
>     [   30.121666] virt-023 corosync[773]:   [KNET  ] pmtud: PMTUD link change for host: 1 link: 0 from 1157 to 1365
>     [   30.122035] virt-023 corosync[773]:   [KNET  ] pmtud: Global data MTU changed to: 1365


Expected results:

According to corosync developer, pacemaker should attempt to enable the corosync cfg shutdown tracker in a retry loop (see the patch: https://github.com/ClusterLabs/pacemaker/pull/2611).

It is not clear whether the retry loop would delay pacemaker crash by a sufficient amount of time for systemd service restart rate-limit to not be hit.


Additional info:

For other unforeseen edge-cases where pacemaker might be crashing "too fast" on service startup, it might be beneficial to explicitly set pacemaker.service's `RestartSec`, `DefaultStartLimitBurst` and `StartLimitBurst` to a carefully selected combination of values.

--- Additional comment from Ken Gaillot on 2022-01-19 15:18:12 UTC ---

Fixed upstream as of commit e8bf0161

<snip>

Comment 4 Patrik Hagara 2022-02-01 12:46:42 UTC

env: 2-node cluster with qdevice/qnetd in lms mode with a slow (eg. `sleep 10`) heuristics configured

reproducer:
1. pcs cluster stop node2 --wait
2. pcs cluster start node2 --wait


before fix (pacemaker-2.1.2-2.el8)
==================================

> [root@virt-058 ~]# journalctl -u corosync -u corosync-qdevice -u pacemaker -f
> Feb 01 13:19:22 virt-058 systemd[1]: Starting Corosync Cluster Engine...
> Feb 01 13:19:22 virt-058 corosync[113248]:   [MAIN  ] Corosync Cluster Engine 3.1.5 starting up
> Feb 01 13:19:22 virt-058 corosync[113248]:   [MAIN  ] Corosync built-in features: dbus systemd xmlconf vqsim nozzle snmp pie relro bindnow
> Feb 01 13:19:22 virt-058 corosync[113248]:   [TOTEM ] Initializing transport (Kronosnet).
> Feb 01 13:19:22 virt-058 corosync[113248]:   [TOTEM ] totemknet initialized
> Feb 01 13:19:22 virt-058 corosync[113248]:   [KNET  ] common: crypto_nss.so has been loaded from /usr/lib64/kronosnet/crypto_nss.so
> Feb 01 13:19:22 virt-058 corosync[113248]:   [SERV  ] Service engine loaded: corosync configuration map access [0]
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QB    ] server name: cmap
> Feb 01 13:19:22 virt-058 corosync[113248]:   [SERV  ] Service engine loaded: corosync configuration service [1]
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QB    ] server name: cfg
> Feb 01 13:19:22 virt-058 corosync[113248]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QB    ] server name: cpg
> Feb 01 13:19:22 virt-058 corosync[113248]:   [SERV  ] Service engine loaded: corosync profile loading service [4]
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QUORUM] Using quorum provider corosync_votequorum
> Feb 01 13:19:22 virt-058 corosync[113248]:   [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QB    ] server name: votequorum
> Feb 01 13:19:22 virt-058 corosync[113248]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QB    ] server name: quorum
> Feb 01 13:19:22 virt-058 corosync[113248]:   [TOTEM ] Configuring link 0
> Feb 01 13:19:22 virt-058 corosync[113248]:   [TOTEM ] Configured link number 0: local addr: 2620:52:0:25a4:1800:ff:fe00:3a, port=5405
> Feb 01 13:19:22 virt-058 corosync[113248]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:19:22 virt-058 corosync[113248]:   [KNET  ] host: host: 1 has no active links
> Feb 01 13:19:22 virt-058 corosync[113248]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:19:22 virt-058 corosync[113248]:   [KNET  ] host: host: 1 has no active links
> Feb 01 13:19:22 virt-058 corosync[113248]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:19:22 virt-058 corosync[113248]:   [KNET  ] host: host: 1 has no active links
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QUORUM] Sync members[1]: 2
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QUORUM] Sync joined[1]: 2
> Feb 01 13:19:22 virt-058 corosync[113248]:   [TOTEM ] A new membership (2.43) was formed. Members joined: 2
> Feb 01 13:19:22 virt-058 corosync[113248]:   [QUORUM] Members[1]: 2
> Feb 01 13:19:22 virt-058 corosync[113248]:   [MAIN  ] Completed service synchronization, ready to provide service.
> Feb 01 13:19:22 virt-058 systemd[1]: Started Corosync Cluster Engine.
> Feb 01 13:19:22 virt-058 systemd[1]: Starting Corosync Qdevice daemon...
> Feb 01 13:19:24 virt-058 corosync[113248]:   [KNET  ] rx: host: 1 link: 0 is up
> Feb 01 13:19:24 virt-058 corosync[113248]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:19:24 virt-058 corosync[113248]:   [KNET  ] pmtud: PMTUD link change for host: 1 link: 0 from 1157 to 1365
> Feb 01 13:19:24 virt-058 corosync[113248]:   [KNET  ] pmtud: Global data MTU changed to: 1365
> Feb 01 13:19:24 virt-058 corosync[113248]:   [QUORUM] Sync members[2]: 1 2
> Feb 01 13:19:24 virt-058 corosync[113248]:   [QUORUM] Sync joined[1]: 1
> Feb 01 13:19:24 virt-058 corosync[113248]:   [TOTEM ] A new membership (1.47) was formed. Members joined: 1
> Feb 01 13:19:35 virt-058 systemd[1]: Started Corosync Qdevice daemon.
> Feb 01 13:19:35 virt-058 systemd[1]: Started Pacemaker High Availability Cluster Manager.
> Feb 01 13:19:35 virt-058 pacemakerd[113270]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
> Feb 01 13:19:35 virt-058 pacemakerd[113270]:  notice: Starting Pacemaker 2.1.2-2.el8
> Feb 01 13:19:35 virt-058 pacemakerd[113270]:  crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Main process exited, code=exited, status=76/PROTOCOL
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Service RestartSec=100ms expired, scheduling restart.
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Scheduled restart job, restart counter is at 1.
> Feb 01 13:19:35 virt-058 systemd[1]: Stopped Pacemaker High Availability Cluster Manager.
> Feb 01 13:19:35 virt-058 systemd[1]: Started Pacemaker High Availability Cluster Manager.
> Feb 01 13:19:35 virt-058 pacemakerd[113273]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
> Feb 01 13:19:35 virt-058 pacemakerd[113273]:  notice: Starting Pacemaker 2.1.2-2.el8
> Feb 01 13:19:35 virt-058 pacemakerd[113273]:  crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Main process exited, code=exited, status=76/PROTOCOL
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Failed with result 'exit-code'.
> Feb 01 13:19:35 virt-058 corosync[113248]:   [QUORUM] This node is within the primary component and will provide service.
> Feb 01 13:19:35 virt-058 corosync[113248]:   [QUORUM] Members[2]: 1 2
> Feb 01 13:19:35 virt-058 corosync[113248]:   [MAIN  ] Completed service synchronization, ready to provide service.
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Service RestartSec=100ms expired, scheduling restart.
> Feb 01 13:19:35 virt-058 systemd[1]: pacemaker.service: Scheduled restart job, restart counter is at 2.
> Feb 01 13:19:35 virt-058 systemd[1]: Stopped Pacemaker High Availability Cluster Manager.
> Feb 01 13:19:35 virt-058 systemd[1]: Started Pacemaker High Availability Cluster Manager.
> Feb 01 13:19:35 virt-058 pacemakerd[113275]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
> Feb 01 13:19:35 virt-058 pacemakerd[113275]:  notice: Starting Pacemaker 2.1.2-2.el8
> Feb 01 13:19:35 virt-058 pacemakerd[113275]:  notice: Pacemaker daemon successfully started and accepting connections

Result: pacemaker.service fails 2 times with "crit: Could not enable Corosync CFG shutdown tracker: CS_ERR_TRY_AGAIN" while corosync is waiting for corosync to establish quorum.


after fix (pacemaker-2.1.2-3.el8)
=================================

> [root@virt-569 ~]# journalctl -u corosync -u corosync-qdevice -u pacemaker -f
> Feb 01 13:39:47 virt-569 systemd[1]: Starting Corosync Cluster Engine...
> Feb 01 13:39:47 virt-569 corosync[126292]:   [MAIN  ] Corosync Cluster Engine 3.1.5 starting up
> Feb 01 13:39:47 virt-569 corosync[126292]:   [MAIN  ] Corosync built-in features: dbus systemd xmlconf vqsim nozzle snmp pie relro bindnow
> Feb 01 13:39:47 virt-569 corosync[126292]:   [TOTEM ] Initializing transport (Kronosnet).
> Feb 01 13:39:47 virt-569 corosync[126292]:   [TOTEM ] totemknet initialized
> Feb 01 13:39:47 virt-569 corosync[126292]:   [KNET  ] common: crypto_nss.so has been loaded from /usr/lib64/kronosnet/crypto_nss.so
> Feb 01 13:39:47 virt-569 corosync[126292]:   [SERV  ] Service engine loaded: corosync configuration map access [0]
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QB    ] server name: cmap
> Feb 01 13:39:47 virt-569 corosync[126292]:   [SERV  ] Service engine loaded: corosync configuration service [1]
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QB    ] server name: cfg
> Feb 01 13:39:47 virt-569 corosync[126292]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QB    ] server name: cpg
> Feb 01 13:39:47 virt-569 corosync[126292]:   [SERV  ] Service engine loaded: corosync profile loading service [4]
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QUORUM] Using quorum provider corosync_votequorum
> Feb 01 13:39:47 virt-569 corosync[126292]:   [SERV  ] Service engine loaded: corosync vote quorum service v1.0 [5]
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QB    ] server name: votequorum
> Feb 01 13:39:47 virt-569 corosync[126292]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1 [3]
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QB    ] server name: quorum
> Feb 01 13:39:47 virt-569 corosync[126292]:   [TOTEM ] Configuring link 0
> Feb 01 13:39:47 virt-569 corosync[126292]:   [TOTEM ] Configured link number 0: local addr: 2620:52:0:25a4:1800:ff:fe00:239, port=5405
> Feb 01 13:39:47 virt-569 corosync[126292]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:39:47 virt-569 corosync[126292]:   [KNET  ] host: host: 1 has no active links
> Feb 01 13:39:47 virt-569 corosync[126292]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:39:47 virt-569 corosync[126292]:   [KNET  ] host: host: 1 has no active links
> Feb 01 13:39:47 virt-569 corosync[126292]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:39:47 virt-569 systemd[1]: Started Corosync Cluster Engine.
> Feb 01 13:39:47 virt-569 corosync[126292]:   [KNET  ] host: host: 1 has no active links
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QUORUM] Sync members[1]: 2
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QUORUM] Sync joined[1]: 2
> Feb 01 13:39:47 virt-569 corosync[126292]:   [TOTEM ] A new membership (2.2d) was formed. Members joined: 2
> Feb 01 13:39:47 virt-569 corosync[126292]:   [QUORUM] Members[1]: 2
> Feb 01 13:39:47 virt-569 corosync[126292]:   [MAIN  ] Completed service synchronization, ready to provide service.
> Feb 01 13:39:47 virt-569 systemd[1]: Starting Corosync Qdevice daemon...
> Feb 01 13:39:49 virt-569 corosync[126292]:   [KNET  ] rx: host: 1 link: 0 is up
> Feb 01 13:39:49 virt-569 corosync[126292]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
> Feb 01 13:39:49 virt-569 corosync[126292]:   [KNET  ] pmtud: PMTUD link change for host: 1 link: 0 from 1157 to 1365
> Feb 01 13:39:49 virt-569 corosync[126292]:   [KNET  ] pmtud: Global data MTU changed to: 1365
> Feb 01 13:39:50 virt-569 corosync[126292]:   [QUORUM] Sync members[2]: 1 2
> Feb 01 13:39:50 virt-569 corosync[126292]:   [QUORUM] Sync joined[1]: 1
> Feb 01 13:39:50 virt-569 corosync[126292]:   [TOTEM ] A new membership (1.31) was formed. Members joined: 1
> Feb 01 13:40:00 virt-569 systemd[1]: Started Corosync Qdevice daemon.
> Feb 01 13:40:00 virt-569 systemd[1]: Started Pacemaker High Availability Cluster Manager.
> Feb 01 13:40:00 virt-569 pacemakerd[126314]:  notice: Additional logging available in /var/log/pacemaker/pacemaker.log
> Feb 01 13:40:00 virt-569 pacemakerd[126314]:  notice: Starting Pacemaker 2.1.2-3.el8
> Feb 01 13:40:00 virt-569 corosync[126292]:   [QUORUM] This node is within the primary component and will provide service.
> Feb 01 13:40:00 virt-569 corosync[126292]:   [QUORUM] Members[2]: 1 2
> Feb 01 13:40:00 virt-569 corosync[126292]:   [MAIN  ] Completed service synchronization, ready to provide service.
> Feb 01 13:40:01 virt-569 pacemakerd[126314]:  notice: Pacemaker daemon successfully started and accepting connections

Result: pacemaker.service does not exit with a fatal error while corosync is establishing quorum, but patiently retries.

Comment 6 errata-xmlrpc 2022-05-10 14:10:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pacemaker bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1885