Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem: While considering and developing guidance for customers to use in deploying corosync-qdevice as a quorum arbitration method in their clusters, it is becoming clear that the reliability of the corosync-qnetd server can become a significant concern.
When choosing an algorithm, the main consideration is whether the qdevice should have great or only minor voting power. Giving the qdevice great power (lms) means the cluster can survive more node failures, but means the cluster is more vulnerable to outages if the qnetd connection goes away. Giving the cluster has more chance to survive if the qnetd connection is lost, but the cluster is still susceptible to losses of quorum in wider-scale node failures.
If you want your cluster to survive as many failure scenarios as possible, then neither of these choices is entirely ideal.
However, if we can enhance the reliability and recoverability of the qnetd server, suddenly lms offers the best of both worlds - ability to survive wide-scale failures and also minimized possibility of losing quorum due to loss of qnetd.
I am filing this request for us to consider possible enhancements to corosync-qnetd and corosync-qdevice that can improve reliability of the net model to minimize it as a point of failure.
In internal discussions, it has been made clear that we're already considering the idea of being able to cluster the corosync-qnetd process, which would greatly help in making the qdevice more reliable. If that is already underway, I'd like to use this bug report to track those efforts and target an upcoming release.
An additional possible enhancement I haven't heard mentioned yet is the potential to use redundant interconnects between qdevice and qnetd. Even if we offer a clustered qnetd, we're still susceptible to network outages eliminating any further fault-tolerance of the cluster. A qnetd-network-outage + a node failure would typically spell a complete loss of quorum throughout the cluster. If that is a reasonable prospect, then I'd like to either track that here as well or have us split another report to track it.
Let me know if there are any ideas, concerns, or questions.
Thanks,
John
(In reply to John Ruemker from comment #0)
> Giving the cluster has more chance to survive if the qnetd connection is
> lost, but the cluster is still susceptible to losses of quorum in
> wider-scale node failures.
>
Giving the *qdevice less voting power (ffsplit) means* the cluster has more chance to survive if the qnetd connection is lost, but the cluster is still susceptible to loss of quorum in wider-scale node failures.
Yes, I agree ideal solution would be to have all options you mentioned so:
- HA (active/passive) - that should work even today it's just matter of testing
- redundant interconnects
I would just add last option:
- "unified" algorithm where it's possible to define 1 - (n - 1) votes
Closing this BZ for now. Ideas in here are important to implement, but it's not trivial (and not too much fun to keep moving it forward because of other priorities). It's also more upstream than downstream, so created GH issue https://github.com/corosync/corosync-qdevice/issues/23 to track progress.