Bug 1102430

Summary:	ActiveMQ routing plug-in should take array of hosts
Product:	OpenShift Container Platform	Reporter:	Miciah Dashiel Butler Masters <mmasters>
Component:	Node	Assignee:	Miheer Salunke <misalunk>
Status:	CLOSED ERRATA	QA Contact:	libra bugs <libra-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.1.0	CC:	adellape, bleanhar, jialiu, jokerman, libra-onpremise-devel, misalunk, mmccomas, pep, pruan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	When the sample routing plug-in was configured with an array of ActiveMQ brokers, the broker service would not start properly. This bug fix adds support to interpret the ACTIVEMQ_HOST parameter in the /etc/openshift/plugins.d/openshift-origin-routing-activemq.conf file as a comma-separated list of host:port pairs, and use this list of hosts when connecting to ActiveMQ for redundancy. The broker service now starts properly, and the sample routing plug-in can now be configured with more than one ActiveMQ broker.	Story Points:	---
Clone Of:
Clones:	1102432 (view as bug list)		Environment:
Last Closed:	2014-06-23 07:38:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1102432
Bug Blocks:	1109052

Description Miciah Dashiel Butler Masters 2014-05-29 00:21:01 UTC

Description of problem:

The ActiveMQ routing plug-in cannot be configured with more than one ActiveMQ broker.

How reproducible:

Readily.

Steps to Reproduce:

1. Install a fresh OpenShift Enterprise 2.0 or 2.1 PaaS with multiple ActiveMQ brokers.

2. Install and configure the ActiveMQ routing plug-in on the OpenShift broker and ActiveMQ broker hosts per Procedure 7.20 "To Enable and Configure the Sample Routing Plug-in" in the Deployment Guide: https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/2/html-single/Deployment_Guide/index.html#sect-Using_an_External_Load_Balancer_for_High-Availability_Applications

3. Run a listener on the routing plug-in's topic and create, delete, scale up, or scale down an application using rhc.

4. On each OpenShift broker host, edit /etc/openshift/plugins.d/openshift-origin-routing-activemq.conf by changing ACTIVEMQ_HOST to include multiple hosts; for example, ACTIVEMQ_HOST='10.0.0.7:61613,10.0.0.8:61613,10.0.0.9:61613'.

5. On each OpenShift broker host, restart broker: `service openshift-broker restart`.

6. Tail /var/log/openshift/broker/httpd/error_log.

7. Run a listener on the routing plug-in's topic and create, delete, scale up, or scale down an application using rhc.

Actual results:

After Steps 1 through 3, the routing plug-in works fine, and application create/delete/scaling work fine.

At Step 6, there are errors:

connect to 10.0.0.7:61613,10.0.0.8:61613,10.0.0.9:61613 failed: getaddrinfo: Name or service not known will retry(#0) in 5

This error repeats every 5 seconds with the retry # incremented.

At Step 7, rhc hangs for several minutes and then reports an error:

# rhc -k cartridge scale php -a testapp1 3
You have not yet configured the OpenShift client tools. Please run 'rhc setup'.
An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift server 'https://broker01.hosts.example.com/broker/rest/application/53866ea62c165f5769000037/cartridge/php-5.3'.

Expected results:

There should be no errors at Step 6, and Step 7 should work just as well as Step 3.

Additional info:

This bug impacts HA.

Comment 1 Miciah Dashiel Butler Masters 2014-05-29 00:25:06 UTC

Waiting for fix to get merged upstream before cherry-picking.

Comment 4 Miheer Salunke 2014-06-10 17:57:11 UTC

PR https://github.com/openshift/enterprise-server/pull/297

Merged into Openshift Enterprise 2.0: https://github.com/openshift/enterprise-server/commit/19988d2a323f1c6af9bdba5d453c14739f7e9c18

Comment 5 Miheer Salunke 2014-06-10 18:06:07 UTC

Sorry I meant "Merged into Openshift Enterprise 2.1, not Openshift Enterprise 2.0"

Comment 8 Johnny Liu 2014-06-12 17:13:07 UTC

Verified this bug with 2.1.z/2014-06-11.1, and PASS.

Reproduce this bug on 2.1.z/2014-06-10.3, the same behavior as the initial report.
In /var/log/openshift/broker/httpd/error_log, get the following error:
connect to node2.ose21z-manual.com.cn,broker.ose21z-manual.com.cn failed: getaddrinfo: Name or service not known will retry(#202) in 5

At the client side, get the following error:
rhc app create scaphp54app php-5.4 -p redhat -l jialiu -s
An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift
server 'https://10.3.15.45/broker/rest/api'.


Then upgrade system to rubygem-openshift-origin-routing-activemq-0.4.1.1-1.el6op.noarch, restart openshift-broker service, app could be created successfully, and listener.rb could recived add_public_endpoint activemq message.

Comment 10 errata-xmlrpc 2014-06-23 07:38:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0781.html