Description of problem:

Background requests made to the broker from the console are performed under a hard coded timeout. A customer recently hit this timeout when requesting the /console/applications page. The user hitting the timeout was a member of many domains, each with many gears. This high number of gears/domains appears to be causing the background requests to take over 10 seconds.

Below is the error received:
-=~~~~~~~~~~~~~~~~~~~~~~~~~~=-
2014-09-08 15:55:09.410 [INFO ] Completed 500 Internal Server Error in 10002ms (pid:5886)
2014-09-08 15:55:09.414 [FATAL] AsyncAware::ThreadTimedOut (The thread #<Thread:0x007f605c2997f8 dead> (index=1) did not complete within 10 seconds.):
  openshift-origin-console (1.23.4.4) app/controllers/async_aware.rb:38:in `block in join'
  openshift-origin-console (1.23.4.4) app/controllers/async_aware.rb:34:in `map'
  openshift-origin-console (1.23.4.4) app/controllers/async_aware.rb:34:in `join'
  openshift-origin-console (1.23.4.4) app/controllers/async_aware.rb:47:in `join!'
  openshift-origin-console (1.23.4.4) app/controllers/applications_controller.rb:85:in `index'
-=~~~~~~~~~~~~~~~~~~~~~~~~~~=-

The code that was hit:
-=~~~~~~~~~~~~~~~~~~~~~~~~~~=-
  def index
    if params[:test]
      @applications = Fixtures::Applications.list
      @domains = Fixtures::Applications.list_domains
      (@applications + @domains).each{ |d| d.send(:api_identity_id=, '2') }
    else
      async{ @applications = Application.find :all, :as => current_user, :params => {:include => :cartridges} }
      async{ @domains = user_domains }
-->    join!(10)
    end
-=~~~~~~~~~~~~~~~~~~~~~~~~~~=-

The 10 second timeout is hardcoded. Usually, if this timeout is hit, there is an issue with the broker, which navigating to the broker endpoints (such as /broker/rest/domain//applications?include=cartridges) would have revealed. In this particular case, there were no errors on the broker side. The logs are clean and all endpoints (eventually) return successfully. The call simply takes longer due to the high number of domains/gears that must be presented.

Another factor that we believe is causing this, is breaking up an openshift environment into numerous datacenters, which is now supported. This customer in particular splits their environment across 2 datacenters across the US. If the broker ends up making an mcollective request to one of several activemq instances that are in another datacenter, these broker calls could take even longer.

Version-Release number of selected component (if applicable):
2.1