Description of problem:

While a database transaction is open, RHQ can hold the transaction a long time waiting for the agent to reply to the requestFullAvailabilityReport request. This can be up to 5 minutes. This can cause database locking issues. Stack trace:

"http-/0.0.0.0:7080-53" daemon prio=10 tid=0x00007f3314064000 nid=0x6d22 waiting on condition [0x00007f32627e2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000042c0e69e8> (a java.util.concurrent.FutureTask$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227)
        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
        at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.run(ClientCommandSenderTask.java:143)
        at org.rhq.enterprise.communications.command.client.ClientCommandSender.sendSynch(ClientCommandSender.java:647)
        at org.rhq.enterprise.communications.command.client.ClientRemotePojoFactory$RemotePojoProxyHandler.invoke(ClientRemotePojoFactory.java:418)
        at $Proxy1587.requestFullAvailabilityReport(Unknown Source)
        at org.rhq.enterprise.server.resource.ResourceManagerBean.enableResources(ResourceManagerBean.java:2885)

The server should send the request asynchronously and never block. Ideally, the request could be made in a separate thread as well.

Similar issues exist in other methods: Uninventory comes to mind.


    @Override
    @TransactionAttribute(TransactionAttributeType.NEVER)
    public List<Integer> enableResources(Subject subject, int[] resourceIds) {
        // On a best effort basic, ask the relevant agents that their next avail report be full, so that we get
        // the current avail type for the newly enabled resources.  If we can't contact the agent don't worry about
        // it; if it's down we'll get a full report when it comes up.
        // TODO: This may need to be made out of band if perf becomes an issue.
        for (Agent agent : reports.keySet()) {

Version-Release number of selected component (if applicable): 4.9


How reproducible: Always.


Steps to Reproduce:
1. Create an agent that is running but does not reply to requests. (Basically create an agent that's running on the same port but won't reply to the server.) 
2. Attempt to enableResources on this agent.
3. Watch the thread hang.

Actual results: Thread hangs, database transactions hang.


Expected results: Should exit immediately.


Additional info: