RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 684825 - [PATCH] Fix pacemaker's wrong quorum view in a CMAN+pacemaker cluster
Summary: [PATCH] Fix pacemaker's wrong quorum view in a CMAN+pacemaker cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: pacemaker
Version: 6.0
Hardware: All
OS: Linux
urgent
medium
Target Milestone: rc
: 6.1
Assignee: Andrew Beekhof
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-03-14 15:36 UTC by Simone Gotti
Modified: 2011-08-12 12:48 UTC (History)
3 users (show)

Fixed In Version: pacemaker-1.1.5-3.el6
Doc Type: Technology Preview
Doc Text:
In a cluster environment managed by both Pacemaker and the CMAN cluster management subsystem, frequent leaving and joining of a node could cause Pacemaker's quorum view to be incorrect. This update applies a patch that addresses this issue, so that the leaving and joining of a node no longer causes Pacemaker's quorum view to be different from CMAN's.
Clone Of:
Environment:
Last Closed: 2011-05-19 13:49:39 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Patch to deque all the cman events from crmd (840 bytes, patch)
2011-03-14 15:36 UTC, Simone Gotti 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0642 0 normal SHIPPED_LIVE pacemaker bug fix and enhancement update 2011-05-18 17:55:40 UTC

Description Simone Gotti 2011-03-14 15:36:21 UTC 
Created attachment 484227 [details]
Patch to deque all the cman events from crmd

Testing a cman+pacemaker cluster on rhel6 I noticed a very nasty
behavior when some nodes were leaving and rejoining the cluster. When a
nodes starts leaving and rejoining the cluster the quorum view of
pacemaker starts becoming sometimes different from the quorum view of
cman. The one not telling the truth was pacemaker.

I reproduced the problem with a simple test case made of 2 nodes using
cman (no two_nodes flag) and pacemaker (started only on the first node:
pcmk01).

For the tests I was using the latest version of pacemaker (1.1.5) while
keeping the original versions of corosync and cluster (cman) packages
provided by the rhel6 (corosync-1.2.3-21.el6.x86_64,
cman-3.0.12-23.el6.4.x86_64)

The problem is that when a node joins a cluster (starting cman) the cman
on the other nodes emits not one but 2 events (I didn't investigated if
this is normal or present only in some versions of cman) but when crmd
calls cman_dispatch it's using the flag CMAN_DISPATCH_ONE so only one of
the two events is dequeued. In the subsequent cluster event the old one
is dequeued.

The fix I tried used CMAN_DISPATCH_ALL instead of CMAN_DISPATCH_ONE and
looks like its working.

I'm CCing the cluster-devel list as they can be interested in the double
event emitted by cman.


Thanks.

Bye!


== Test case ==

=== Without the patch ===

Start with both nodes with cman started (so the cluster is quorate).


Now stop cman on pcmk02. Output on pcmk01:

pcmk01 corosync[16793]:   [CMAN  ] quorum lost, blocking activity
pcmk01 corosync[16793]:   [QUORUM] This node is within the non-primary
component and will NOT provide any services.
pcmk01 corosync[16793]:   [QUORUM] Members[1]: 1
pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
membership and a new membership was formed.
pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 1
pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
ip(192.168.200.71)
pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
ready to provide service.
pcmk01 crmd: [16993]: notice: cman_event_callback: Membership 668:
quorum lost

Only one event is enqueued.

Now start again cman on pcmk02. Output on pcmk01:

pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
membership and a new membership was formed.
pcmk01 corosync[16793]:   [CMAN  ] quorum regained, resuming activity
pcmk01 corosync[16793]:   [QUORUM] This node is within the primary
component and will provide service.
pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
pcmk01 crmd: [16993]: notice: cman_event_callback: Membership 672:
quorum acquired
pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
ip(192.168.200.71)
pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
ready to provide service.

As you can see two events are enqueued and only one si dequeued (due to
the CMAN_DISPATCH_ONE flag passed to cman_dispatch).

The quorum is regained both on cman and crmd. But there's another event
saying that the quorum is regained in the queue.


Now stop again cman on pcmk02. Output on pcmk01:

pcmk01 corosync[16793]:   [CMAN  ] quorum lost, blocking activity
pcmk01 corosync[16793]:   [QUORUM] This node is within the non-primary
component and will NOT provide any services.
pcmk01 corosync[16793]:   [QUORUM] Members[1]: 1
pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
membership and a new membership was formed.
pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 1
pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
ip(192.168.200.71)
pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
ready to provide service.
pcmk01 crmd: [16993]: info: cman_event_callback: Membership 676: quorum
retained

CMAN says that the quorum is lost and only one event is dispatched. But
crmd dequeued the previous event and thinks that we have the quorum.


Now start again cman on pcmk02. Output on pcmk01:

pcmk01 corosync[16793]:   [TOTEM ] A processor joined or left the
membership and a new membership was formed.
pcmk01 corosync[16793]:   [CMAN  ] quorum regained, resuming activity
pcmk01 corosync[16793]:   [QUORUM] This node is within the primary
component and will provide service.
pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
pcmk01 corosync[16793]:   [QUORUM] Members[2]: 1 2
pcmk01 crmd: [16993]: notice: cman_event_callback: Membership 680:
quorum lost
pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
pcmk01 corosync[16793]:   [CPG   ] downlist received left_list: 0
pcmk01 corosync[16793]:   [CPG   ] chosen downlist from node r(0)
ip(192.168.200.71)
pcmk01 corosync[16793]:   [MAIN  ] Completed service synchronization,
ready to provide service.

CMAN says that the quorum is regained but crmd dequeued again the old
event and now it says that the quorum is lost. And so on...



=== With the patch ===

stop cman on pcmk02. Output on pcmk01:

pcmk01 corosync[13149]:   [CMAN  ] quorum lost, blocking activity
pcmk01 corosync[13149]:   [QUORUM] This node is within the non-primary
component and will NOT provide any services.
pcmk01 corosync[13149]:   [QUORUM] Members[1]: 1
pcmk01 corosync[13149]:   [TOTEM ] A processor joined or left the
membership and a new membership was formed.
pcmk01 corosync[13149]:   [CPG   ] downlist received left_list: 1
pcmk01 corosync[13149]:   [CPG   ] chosen downlist from node r(0)
ip(192.168.200.71)
pcmk01 corosync[13149]:   [MAIN  ] Completed service synchronization,
ready to provide service.

 pcmk01 crmd: [13351]: notice: cman_event_callback: Membership 648:
quorum lost

Only one event is enqueued.


Now start again cman on pcmk02. Output on pcmk01:

pcmk01 corosync[13149]:   [TOTEM ] A processor joined or left the
membership and a new membership was formed.
pcmk01 corosync[13149]:   [CMAN  ] quorum regained, resuming activity
pcmk01 corosync[13149]:   [QUORUM] This node is within the primary
component and will provide service.
pcmk01 corosync[13149]:   [QUORUM] Members[2]: 1 2
pcmk01 corosync[13149]:   [QUORUM] Members[2]: 1 2
pcmk01 crmd: [13351]: notice: cman_event_callback: Membership 652:
quorum acquired
pcmk01 corosync[13149]:   [CPG   ] downlist received left_list: 0
pcmk01 corosync[13149]:   [CPG   ] downlist received left_list: 0
pcmk01 corosync[13149]:   [CPG   ] chosen downlist from node r(0)
ip(192.168.200.71)
pcmk01 corosync[13149]:   [MAIN  ] Completed service synchronization,
ready to provide service.
pcmk01 crmd: [13351]: info: cman_event_callback: Membership 652: quorum
retained

As you can see two events are enqueued and both are dequeued.
Comment 2 Andrew Beekhof 2011-03-14 15:55:50 UTC 
Its very important that we have an up-to-date view of membership/quorum.
The attached patch is correct.
Comment 5 Jaromir Hradilek 2011-04-26 14:47:11 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
In a cluster environment managed by both Pacemaker and the CMAN cluster management subsystem, frequent leaving and joining of a node could cause Pacemaker's quorum view to be incorrect. This update applies a patch that addresses this issue, so that the leaving and joining of a node no longer causes Pacemaker's quorum view to be different from CMAN's.
Comment 6 errata-xmlrpc 2011-05-19 13:49:39 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0642.html
Note You need to log in before you can comment on or make changes to this bug.