Bug 544306 - clustered broker does not retry CPG calls that return TRY_AGAIN
Summary: clustered broker does not retry CPG calls that return TRY_AGAIN
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: 1.2
Hardware: All
OS: Linux
medium
medium
Target Milestone: 1.3
: ---
Assignee: mick
QA Contact: MRG Quality Engineering
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-12-04 14:49 UTC by Alan Conway
Modified: 2011-08-12 16:22 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Very infrequently, adding a new broker to a cluster failed with the following error message: critical Unexpected error: Cannot join CPG group When this occurred, attempting to add the new broker again usually resulted in success. This failure was caused by a missing checks in the CPG clustering-substrate code that have now been added, with the result that new brokers can be successfully added to a cluster on the first try.
Clone Of:
Environment:
Last Closed: 2010-10-14 16:14:26 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0773 0 normal SHIPPED_LIVE Moderate: Red Hat Enterprise MRG Messaging and Grid Version 1.3 2010-10-14 15:56:44 UTC

Description Alan Conway 2009-12-04 14:49:36 UTC
Description of problem:

https://issues.apache.org/jira/browse/QPID-2235

Causes occasional failures on joining a cluster like this:

soak-22: 2009-12-03 15:49:44 notice Initializing CPG
soak-22: 2009-12-03 15:49:44 critical Unexpected error: Cannot join CPG group soakTestCluster_9edd905b-92b3-4cfb-803f-120d7a088f1f: try again (6)

Comment 1 mick 2009-12-11 15:32:55 UTC
fixed with svn commit 889657

Comment 5 mick 2010-10-05 14:12:27 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:  Adding a new broker to a cluster fails about 2% of the time with a messa
ge like this: critical Unexpected error: Cannot join CPG group ...

Consequence:  The new broker does not joing the cluster and you have to try agai
n.

Fix: The CPG clustering-substrate code occasionally returns a TRY_AGAIN flag.  Several places in the cluster code did not test for it, and failed rather than re
trying.

Result: This message can no longer be detected in a reasonable number ( 500 ) of new broker-additions to a cluster.

Comment 6 mick 2010-10-05 14:12:59 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,5 +1,4 @@
-Cause:  Adding a new broker to a cluster fails about 2% of the time with a messa
-ge like this: critical Unexpected error: Cannot join CPG group ...
+Cause:  Adding a new broker to a cluster fails about 2% of the time with a message like this: critical Unexpected error: Cannot join CPG group ...
 
 Consequence:  The new broker does not joing the cluster and you have to try agai
 n.

Comment 7 Douglas Silas 2010-10-05 19:00:19 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,9 +1 @@
-Cause:  Adding a new broker to a cluster fails about 2% of the time with a message like this: critical Unexpected error: Cannot join CPG group ...
+When this occurred, attempting to add the new broker again usually resulted in success. This failure was caused by a missing checks in the CPG clustering-substrate code that have now been added, with the result that new brokers can be successfully added to a cluster on the first try.-
-Consequence:  The new broker does not joing the cluster and you have to try agai
-n.
-
-Fix: The CPG clustering-substrate code occasionally returns a TRY_AGAIN flag.  Several places in the cluster code did not test for it, and failed rather than re
-trying.
-
-Result: This message can no longer be detected in a reasonable number ( 500 ) of new broker-additions to a cluster.

Comment 8 Douglas Silas 2010-10-11 09:47:27 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1,5 @@
+Very infrequently, adding a new broker to a cluster failed with the following error message:
+
+    critical Unexpected error: Cannot join CPG group
+
 When this occurred, attempting to add the new broker again usually resulted in success. This failure was caused by a missing checks in the CPG clustering-substrate code that have now been added, with the result that new brokers can be successfully added to a cluster on the first try.

Comment 10 errata-xmlrpc 2010-10-14 16:14:26 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html


Note You need to log in before you can comment on or make changes to this bug.