Bug 1124619 - Agent Update prompt-command is executed 5 minutes after RHQ Agent operation is invoked
Summary: Agent Update prompt-command is executed 5 minutes after RHQ Agent operation i...
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Agent
Version: JON 3.2.2
Hardware: Unspecified
OS: Unspecified
Target Milestone: ER03
: JON 3.3.0
Assignee: John Mazzitelli
QA Contact: Sunil Kondkar
Depends On:
TreeView+ depends on / blocked
Reported: 2014-07-29 23:14 UTC by Larry O'Leary
Modified: 2018-12-06 17:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Using the Agent prompt --update command resulted in a delay. The update action did not occur until several minutes after the --update command was executed. The --update command has been replaced with an Update Agent operation. The update operation is executed immediately on the remote agent.
Clone Of:
Last Closed: 2014-12-11 14:02:06 UTC
Type: Bug

Attachments (Terms of Use)

Description Larry O'Leary 2014-07-29 23:14:50 UTC
Description of problem:
If a user invoked the agent prompt command:

    update --update

from the RHQ Agent resource's _Execute Prompt Command_ operation, the command is delayed by 299 seconds:

2014-07-29 17:35:06,952 INFO  [ResourceContainer.invoker.nonDaemon-2] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.prompt-command-invoked}Prompt command invoked: [update, --update]
2014-07-29 17:35:06,952 INFO  [RHQ Agent Update Thread] (org.rhq.enterprise.agent.AgentUpdateThread)- {AgentUpdateThread.started}The agent update thread has started - will begin the agent auto-update now!
2014-07-29 17:35:06,953 INFO  [RHQ Agent Update Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.shutting-down}Agent is being shut down...
2014-07-29 17:35:06,953 INFO  [RHQ Primary Server Switchover Thread] (org.rhq.enterprise.agent.AgentMain)- {PrimaryServerSwitchoverThread.stopped}The primary server switchover thread has stopped.
2014-07-29 17:35:06,953 INFO  [RHQ Agent Update Thread] (org.rhq.core.pc.PluginContainer)- Plugin container is being shutdown...
2014-07-29 17:35:06,953 INFO  [RHQ Agent Update Thread] (org.rhq.core.pc.PluginContainer)- Plugin container shutdown will wait up to 300 seconds for shut down background threads to terminate.
2014-07-29 17:35:06,955 INFO  [RHQ Agent Update Thread] (rhq.core.pc.content.ContentManager)- Shutting down Content Manager...
2014-07-29 17:35:06,957 ERROR [OperationManager.invoker-1] (rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler)- Thread [OperationManager.invoker-1] was interrupted.
2014-07-29 17:35:06,957 WARN  [OperationManager.invoker-1] (rhq.core.pc.inventory.ResourceContainer$ComponentInvocation)- Invocation has been marked interrupted for method [OperationFacet.invokeOperation] on resource [Resource[id=10002, uuid=dfa22364-9154-4532-993e-758876a1f191, type={RHQAgent}RHQ Agent, key=Agent 01164018 on jboss.example.com RHQ Agent, name=RHQ Agent, parent=Agent 01164018 on jboss.example.com, version=4.9.0.JON320GA]]
2014-07-29 17:40:06,954 WARN  [RHQ Agent Update Thread] (org.rhq.core.pc.PluginContainer)- Timed out after [299] seconds while waiting for all threads in pool [OperationManager.invoker] to terminate - [1] threads in the pool are still active.

In the above log excerpt it is clear to see what the issue is. The OperationManager thread is still waiting for the operation to finish. Therefore, it won't shutdown when the agent has asked it to so that the update could proceed. Instead, the agent update thread waits for 299 seconds and then interrupts the thread. Thus, allowing the agent update to proceed.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Install and start JBoss ON 3.2 system.
2. Install remote agent.
3. Import remote RHQ Agent resource into inventory.
4. Apply 3.2.2 to RHQ_SERVER_HOME.
5. Start JBoss ON server.
6. From UI, invoke remote RHQ Agent's _Execute Prompt Command_ operation with the following _Command_ argument:

       update --update

Actual results:
Remote agent does not get updated until over 5 minutes later.

Expected results:
Remote agent should be updated right away.

Additional info:
Perhaps the easiest solution for this is to add a new RHQ Agent operation that exposes the same/similar agent prompt command as a standalone operation that can return fast? This allowing the operation manager to shutdown right away instead of waiting?

Comment 1 Jay Shaughnessy 2014-09-08 14:36:34 UTC
There are some other (non-prompt-command) ops that workaround this problem.  A new op may help get around this, assigning to Mazz, he knows about this stuff.

Comment 2 John Mazzitelli 2014-09-08 20:07:44 UTC
master commit:

commit a894f4becfcaf958d6f92091571fe623bc202fa6
Author: John Mazzitelli <mazz@redhat.com>
Date:   Mon Sep 8 16:06:48 2014 -0400

    BZ 1124619 - provide more responsiveness when you want to ask an agent to update itself from the GUI

this needs to be cherry picked

Comment 3 John Mazzitelli 2014-09-08 20:08:41 UTC
what this now does is provide an actual operation named "Update Agent" - you execute this operation to ask the agent to update itself. You no longer have to go through the "update" prompt command via "Execute Prompt Command" to do this.

Comment 4 John Mazzitelli 2014-09-09 14:14:12 UTC
stefan cherry picked to 3.3. branch: abb016f51c23896f450eeca7acaa78a435f0a5da

Comment 5 Simeon Pinder 2014-09-17 02:49:00 UTC
Moving to ON_QA as available for test with the following brew build:

Comment 6 Sunil Kondkar 2014-09-18 11:43:13 UTC
Verified on version : 3.3.0.ER03 build Number :4aefe39:44e33a4

The agent resource displays 'Update Agent' operation in the drop-down. The operation is executed immediately on the remote agent with success status.

Note You need to log in before you can comment on or make changes to this bug.