Bug 1014294

Description of problem:
The networkConnectors configuration described in Section 5.8.4.4.1. "Configuring a Network of ActiveMQ Brokers" of the Deployment Guide can lead to lost replies from node hosts, resulting in random failures.

The networkConnectors configuration for queues needs to set conduitSubscriptions="false" (the default is conduitSubscriptions="true", and this value for the setting is appropriate for topics alone).

Specifically, the following configuration from the Guide is problematic:

      <networkConnectors>
        <networkConnector name="broker1-broker2" duplex="true"
              uri="static:(tcp://activemq2.example.com:61616)"
              userName="amquser" password="amqpass">
          <staticallyIncludedDestinations>
            <topic physicalName="mcollective.openshift.reply"/>
            <topic physicalName="mcollective.discovery.reply"/>
          </staticallyIncludedDestinations>
        </networkConnector>
        <networkConnector name="broker1-broker3" duplex="true"
              uri="static:(tcp://activemq3.example.com:61616)"
              userName="amquser" password="amqpass">
          <staticallyIncludedDestinations>
            <topic physicalName="mcollective.openshift.reply"/>
            <topic physicalName="mcollective.discovery.reply"/>
          </staticallyIncludedDestinations>
        </networkConnector>
      </networkConnectors>

It should be as follows:

      <networkConnectors>
        <networkConnector name="broker1-broker2-topic" duplex="true"
              uri="static:(tcp://activemq2.example.com:61616)"
              userName="amquser" password="amqpass">
          <excludedDestinations>
            <queue physicalName=">" />
          </excludedDestinations>
        </networkConnector>
        <networkConnector name="broker1-broker2-queue" duplex="true"
              uri="static:(tcp://activemq2.example.com:61616)"
              userName="amquser" password="amqpass"
              conduitSubscriptions="false">
          <excludedDestinations>
            <topic physicalName=">" />
          </excludedDestinations>
        </networkConnector>
        <networkConnector name="broker1-broker3-topic" duplex="true"
              uri="static:(tcp://activemq3.example.com:61616)"
              userName="amquser" password="amqpass">
          <excludedDestinations>
            <queue physicalName=">" />
          </excludedDestinations>
        </networkConnector>
        <networkConnector name="broker1-broker3-queue" duplex="true"
              uri="static:(tcp://activemq3.example.com:61616)"
              userName="amquser" password="amqpass"
              conduitSubscriptions="false">
          <excludedDestinations>
            <topic physicalName=">" />
          </excludedDestinations>
        </networkConnector>
      </networkConnectors>

Note that it is necessary to have two networkConnector stanzas for each remote broker: one for topics (without conduitSubscriptions="false" so it has the default value of conduitSubscriptions="true" as before; queues must be explicitly excluded from this networkConnector stanza) and one for queues (with conduitSubscriptions="true"; topics must be explicitly excluded from this networkConnector stanza).


Version-Release number of selected component (if applicable):

OpenShift Enterprise Deployment Guide Edition 1.2

https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/1/html-single/Deployment_Guide/index.html#sect-Configuring_Redundancy


How reproducible:

The problematic networkConnectors configuration sometimes shows up consistently, sometimes only ~20% of the time in testing, but I have been able to reproduce it on all (multiple) OSE 1.2 installations with ActiveMQ replication where I have tried.


Steps to Reproduce:

1. Follow the steps at https://access.redhat.com/site/documentation/en-US/OpenShift_Enterprise/1/html-single/Deployment_Guide/index.html#sect-Configuring_Redundancy

2. Run `mco facts hostname` on the OpenShift broker host.


Actual results:

Sometimes the broker does not get a response from some nodes, as seen in the following:

    [root@broker01 ~]# mco facts hostname
    Report for fact: hostname
    
            node01                                  found 1 times
            node02                                  found 1 times
            node05                                  found 1 times
    
    Finished processing 3 / 5 hosts in 12007.06 ms
    
    
    No response from:
    
       node03.example.com            node04.example.com
    
    [root@broker01 ~]#


Expected results:

The broker should always get a response from all nodes that are connected to any of the configured ActiveMQ brokers:

    [root@broker01 ~]# mco facts hostname
    Report for fact: hostname
    
            node01                                  found 1 times
            node02                                  found 1 times
            node03                                  found 1 times
            node04                                  found 1 times
            node05                                  found 1 times
    
    Finished processing 5 / 5 hosts in 55.88 ms
    [root@broker01 ~]#

Additional info:

MCollective on the broker sends requests to the node hosts using MCollective messages that go over an ActiveMQ topic, and the node hosts send replies using MCollective messages that go over ActiveMQ queues.  The broker subscribes to these queues using JMS selectors.  However, if conduitSubscriptions is enabled for queues on the ActiveMQ brokers, some of those replies on these queues never reach the broker.

The suggested configuration change is consistent with the example ActiveMQ configuration files that are in the upstream MCollective Github repository:

    https://github.com/puppetlabs/marionette-collective/tree/master/ext/activemq/examples/multi-broker

We have also merged the same configuration changes into the installation script:

    https://github.com/openshift/openshift-extras/blob/59ff4687c82d8bf66726569ccac5600ee561331a/enterprise/install-scripts/openshift.ks#L1488

Following are some excerpts from documentation to support the above analysis:

"MCollective sends replies on uniquely-named, single-use queues with names like mcollective.reply.<UNIQUE ID>.

http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#reply-queue-pruning

"[The mcollective.nodes queue is] used for direct addressing; this is a single destination that uses JMS selectors, rather than a group of destinations[.]"

    http://docs.puppetlabs.com/mcollective/deploy/middleware/activemq.html#topic-and-queue-names

"If your consumers use JMS selectors, you should be aware that some interaction can occur between JMS selectors and conduit subscriptions. In general, JMS selectors are always applied by the consumer's local broker (that is, the broker to which the consumer is directly connected). But if the conduit subscriptions option is enabled (which it is by default), JMS selector information is omitted from any conduit subscriptions transmitted to a remote broker. This implies that the remote broker (or brokers ) do not perform any filtering based on JMS selectors. There are certain scenarios where this behavior can cause problems."

    http://fusesource.com/docs/esb/4.3/amq_clustering/Networks-Filtering.html
    https://access.redhat.com/site/documentation/en-US/Fuse_ESB/4.3.1/html-single/Clustering_Guide/index.html#d0e2133