Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
DescriptionHeinz Mauelshagen
2022-05-11 21:14:59 UTC
Description of problem:
Adding a list of nodes to a cluster causes
"Warning: rhel-9.1-n1: Unable to reload corosync configuration: Unable to reload corosync configuration: Could not reload configuration. Error CS_ERR_TRY_AGAIN"
for member nodes and
"Error: Unable to perform operation on any available node/host, therefore it is not possible to continue
Error: Errors have occurred, therefore pcs is unable to continue"
Version-Release number of selected component (if applicable):
pcs 0.11.1-10.el9, pacemaker 2.1.2-4.el9
How reproducible:
for n in n5 n6 n7 ...;do pcs cluster node add $n --enable --start;done
Steps to Reproduce:
1. create e.g. 3 node cluster
2. prepare additional (virtual) machines to add as nodes
3. run ^ shell loop
Actual results:
see above and in attachments.
Also, nodes have to be rebooted to recover.
Expected results:
bulk node addtions succeeding
Additional info:
see attachments
Comment 2Heinz Mauelshagen
2022-05-12 12:31:50 UTC
As a result of node addition failing and even after fence reboot of problematic ones:
# pcs resource move xfs rhel-9.1-n7
Error: Unable to update cib
Call cib_apply_diff failed (-62): Timer expired
Comment 3Heinz Mauelshagen
2022-05-12 12:39:57 UTC
Also, nodes hang on reboot.
Fond all corosync logs attached.
Comment 4Heinz Mauelshagen
2022-05-12 12:42:00 UTC
Comment 5Heinz Mauelshagen
2022-05-12 14:55:16 UTC
I assume this to be a race as
for n in rhel-9.1-n{4..15}; do pcs cluster node add $n --enable --start; sleep 10; done
works fine for the multiple node addition.
Afterwards, "pcs resource move $res $node" fails triggering fencing until after all nodes got rebooted though.
I'm unable to reproduce this issue. The reproducer works perfectly fine for me with no hiccups.
There's a lot of "host: x has no active links", "PMTUD link change for host" and "Token has not been received in" messages in the attached logs. This may be a corosync, or more likely, a network issue.
Heinz:
Can you provide corosync.conf and CIB of the three-node cluster, i.e. before running the loop which adds nodes? Also, systemctl and corosync logs for "Unable to reload corosync configuration: Could not reload configuration. Error CS_ERR_TRY_AGAIN" from the node where the reload failed may be helpful.
I created a 3-node cluster running a fence_xvm device, an IP address, a group consisting of LVM and filesystem resources, colocated the group with the IP, and set resource-stickiness to 1. Then I repeatedly added (and removed) 9 nodes in a loop as shown in comment 0. Even after several attempts, I was not able to trigger the issue.
Because the errors come from corosync, I'm moving this to corosync for further investigation.
@Heinz:
There are too many "Token has not been received in XXXX ms" - so this is probably the reason for CS_ERR_TRY_AGAIN failure. Also there is no "Corosync main process was not scheduled ..." what means the problem is probably network related.
Is there any rate limiting or packet size limiting in the network? Are you able to reproduce the bug reliably? If so, are you able to reproduce it with less node (lets say 8/12/...)? If your machine allows it, could you try to test same scenario with 16 VM running on one hypervisor machine?