Red Hat Bugzilla – Bug 544306
clustered broker does not retry CPG calls that return TRY_AGAIN
Last modified: 2011-08-12 12:22:10 EDT
Description of problem: https://issues.apache.org/jira/browse/QPID-2235 Causes occasional failures on joining a cluster like this: soak-22: 2009-12-03 15:49:44 notice Initializing CPG soak-22: 2009-12-03 15:49:44 critical Unexpected error: Cannot join CPG group soakTestCluster_9edd905b-92b3-4cfb-803f-120d7a088f1f: try again (6)
fixed with svn commit 889657
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: Adding a new broker to a cluster fails about 2% of the time with a messa ge like this: critical Unexpected error: Cannot join CPG group ... Consequence: The new broker does not joing the cluster and you have to try agai n. Fix: The CPG clustering-substrate code occasionally returns a TRY_AGAIN flag. Several places in the cluster code did not test for it, and failed rather than re trying. Result: This message can no longer be detected in a reasonable number ( 500 ) of new broker-additions to a cluster.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,5 +1,4 @@ -Cause: Adding a new broker to a cluster fails about 2% of the time with a messa -ge like this: critical Unexpected error: Cannot join CPG group ... +Cause: Adding a new broker to a cluster fails about 2% of the time with a message like this: critical Unexpected error: Cannot join CPG group ... Consequence: The new broker does not joing the cluster and you have to try agai n.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,9 +1 @@ -Cause: Adding a new broker to a cluster fails about 2% of the time with a message like this: critical Unexpected error: Cannot join CPG group ... +When this occurred, attempting to add the new broker again usually resulted in success. This failure was caused by a missing checks in the CPG clustering-substrate code that have now been added, with the result that new brokers can be successfully added to a cluster on the first try.- -Consequence: The new broker does not joing the cluster and you have to try agai -n. - -Fix: The CPG clustering-substrate code occasionally returns a TRY_AGAIN flag. Several places in the cluster code did not test for it, and failed rather than re -trying. - -Result: This message can no longer be detected in a reasonable number ( 500 ) of new broker-additions to a cluster.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1,5 @@ +Very infrequently, adding a new broker to a cluster failed with the following error message: + + critical Unexpected error: Cannot join CPG group + When this occurred, attempting to add the new broker again usually resulted in success. This failure was caused by a missing checks in the CPG clustering-substrate code that have now been added, with the result that new brokers can be successfully added to a cluster on the first try.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0773.html