Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1639826

Summary: rabbitmq-cluster agent should fail monitor when node is in minority partition
Product: Red Hat Enterprise Linux 7 Reporter: John Eckersberg <jeckersb>
Component: resource-agentsAssignee: John Eckersberg <jeckersb>
Status: CLOSED ERRATA QA Contact: cluster-qe <cluster-qe>
Severity: urgent Docs Contact: Marek Suchánek <msuchane>
Priority: urgent    
Version: 7.5CC: agk, aherr, cfeist, chjones, cluster-maint, cnewsom, ctowsley, fdinitto, lmiccini, mjuricek, mkrcmari, morazi, oalbrigt, phagara, rzaleski
Target Milestone: rcKeywords: ZStream
Target Release: 7.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: GSSApproved
Fixed In Version: resource-agents-4.1.1-14.el7 Doc Type: If docs needed, set a value
Doc Text:
Previously, monitor action succeeded on all nodes when the RabbitMQ message broker got into a partitioned state without triggering a fencing action. As a consequence, the cluster remained partitioned indefinitely. With this update, monitor action now fails when node is in minority partition. As a result, nodes in minority partition get fenced as expected.
Story Points: ---
Clone Of:
: 1641946 1641948 (view as bug list) Environment:
Last Closed: 2019-08-06 12:01:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1595753    
Bug Blocks: 1641946, 1641948    

Description John Eckersberg 2018-10-16 17:13:06 UTC
It's possible for rabbitmq to get into a partitioned state without triggering a fencing action.  When this happens, the monitor action still succeeds on all nodes, so the cluster remains partitioned indefinitely.

The monitor action should compare (a) the number of nodes pacemaker thinks is running (via RMQ_CRM_ATTR_COOKIE same as the join list), and (b) the number of nodes rabbitmq itself thinks is running.  If the number from rabbit is a minority of the total, fail the monitor so the minority node restarts and rejoins the majority.

Comment 7 Chris Jones 2018-10-22 13:00:48 UTC
OSP QE reports success with a regression run of the hotfix build.

Comment 10 John Eckersberg 2018-11-02 17:45:25 UTC
Regression fix posted: https://github.com/ClusterLabs/resource-agents/pull/1263

Comment 14 errata-xmlrpc 2019-08-06 12:01:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2012