Bug 756155

Summary: Agent Auto-Upgrade Fails for 2.3.1 -> 3.0 CR2
Product: [Other] RHQ Project Reporter: Robert Buck <rbuck>
Component: AgentAssignee: Deon Ballard <dlackey>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: medium    
Version: 3.0.0 Beta1CC: hrupp, lzoubek, mazz
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 755580 Environment:
Last Closed: 2012-06-27 03:12:27 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 755580    
Bug Blocks: 752488    

Description Robert Buck 2011-11-22 20:34:37 UTC
I upgraded the server. When I restart the agent I expected it would have auto-upgraded itself from the running server. Initially I had a running server and agent version 2.3.1. This was found during upgrade testing.

Here are the results:

rbuck@rbucklaptop:~/tmp/testing/agent/rhq-agent/bin$ ./rhq-agent.sh
RHQ 1.3.1.GA [5295] (Wed Feb 24 18:46:23 EST 2010)
This version of the agent is not supported by the server - an agent update must be applied
The agent update thread has started - will begin the agent auto-update now!
The agent will wait for [2] threads to die
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
Exception in thread "RHQ Agent Registration Thread" java.lang.IllegalStateException: The agent is already in the process of updating itself
        at org.rhq.enterprise.agent.AgentUpdateThread.lock(AgentUpdateThread.java:261)
        at org.rhq.enterprise.agent.AgentUpdateThread.updateAgentNow(AgentUpdateThread.java:77)
        at org.rhq.enterprise.agent.AgentMain$1.run(AgentMain.java:1374)
        at java.lang.Thread.run(Thread.java:662)
This version of the agent is not supported by the server - an agent update must be applied
Failed to start the agent
org.rhq.core.clientapi.server.core.AgentNotSupportedException
        at org.rhq.enterprise.agent.AgentMain.waitForServer(AgentMain.java:1526)
        at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:627)
        at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:406)
Downloaded the agent update binary to [/home/rbuck/tmp/testing/agent/rhq-enterprise-agent-4.2.0.JON300.CR2.jar]
Missing the envvar [RHQ_AGENT_JAVA_EXE_FILE_PATH] - will try to find a Java executable to use.
Will use the Java executable [/opt/os_cpu/Linux/x86/java/jdk1.6.0_27/jre/bin/java]
Now executing agent update - if all goes well, this is the last you will hear of this agent: [ProcessExecution: executable=[/opt/os_cpu/Linux/x86/java/jdk1.6.0_27/jre/bin/java], args=[[-jar, /home/rbuck/tmp/testing/agent/rhq-enterprise-agent-4.2.0.JON300.CR2.jar, --pause=20000, --update=/home/rbuck/tmp/testing/agent/rhq-agent]], env-vars=[null], working-dir=[/home/rbuck/tmp/testing/agent], wait=[0], capture-output=[false], kill-on-timeout=[false], executable-is-command=[true]]
rbuck@rbucklaptop:~/tmp/testing/agent/rhq-agent/bin$ ./rhq-agent.sh
RHQ 1.3.1.GA [5295] (Wed Feb 24 18:46:23 EST 2010)
This version of the agent is not supported by the server - an agent update must be applied
The agent update thread has started - will begin the agent auto-update now!
The agent will wait for [2] threads to die
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
Exception in thread "RHQ Agent Registration Thread" java.lang.IllegalStateException: The agent is already in the process of updating itself
        at org.rhq.enterprise.agent.AgentUpdateThread.lock(AgentUpdateThread.java:261)
        at org.rhq.enterprise.agent.AgentUpdateThread.updateAgentNow(AgentUpdateThread.java:77)
        at org.rhq.enterprise.agent.AgentMain$1.run(AgentMain.java:1374)
        at java.lang.Thread.run(Thread.java:662)
This version of the agent is not supported by the server - an agent update must be applied
Failed to start the agent
org.rhq.core.clientapi.server.core.AgentNotSupportedException
        at org.rhq.enterprise.agent.AgentMain.waitForServer(AgentMain.java:1526)
        at org.rhq.enterprise.agent.AgentMain.start(AgentMain.java:627)
        at org.rhq.enterprise.agent.AgentMain.main(AgentMain.java:406)
Downloaded the agent update binary to [/home/rbuck/tmp/testing/agent/rhq-enterprise-agent-4.2.0.JON300.CR2.jar]
Missing the envvar [RHQ_AGENT_JAVA_EXE_FILE_PATH] - will try to find a Java executable to use.
Will use the Java executable [/opt/os_cpu/Linux/x86/java/jdk1.6.0_27/jre/bin/java]
Now executing agent update - if all goes well, this is the last you will hear of this agent: [ProcessExecution: executable=[/opt/os_cpu/Linux/x86/java/jdk1.6.0_27/jre/bin/java], args=[[-jar, /home/rbuck/tmp/testing/agent/rhq-enterprise-agent-4.2.0.JON300.CR2.jar, --pause=20000, --update=/home/rbuck/tmp/testing/agent/rhq-agent]], env-vars=[null], working-dir=[/home/rbuck/tmp/testing/agent], wait=[0], capture-output=[false], kill-on-timeout=[false], executable-is-command=[true]]
rbuck@rbucklaptop:~/tmp/testing/agent/rhq-agent/bin$

Comment 1 Heiko W. Rupp 2011-11-23 08:19:01 UTC
> rbuck@rbucklaptop:~/tmp/testing/agent/rhq-agent/bin$ ./rhq-agent.sh

It looks like we have an issue if the agent is not in background (started from a wrapper) and thus stdin/out/err are not redirected.

Assigning to mazz for further comments.

Perhaps we need to make this more clear in the various docs.

Comment 2 John Mazzitelli 2011-11-23 14:39:45 UTC
(In reply to comment #1)
> > rbuck@rbucklaptop:~/tmp/testing/agent/rhq-agent/bin$ ./rhq-agent.sh
> 
> It looks like we have an issue if the agent is not in background (started from
> a wrapper) and thus stdin/out/err are not redirected.
> 
> Assigning to mazz for further comments.
> 
> Perhaps we need to make this more clear in the various docs.

This is clear in our online docs:

http://rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-StartTheAgentUsingTheWrapperService

Though it looks like we need the JON docs updated:

http://documentation-stage.bne.redhat.com/docs/en-US/JBoss_Operations_Network/3.0/html/Installation_Guide/upgrading-the-agent.html#Preparing_the_Agent

assigning to deon for that

Comment 3 Robert Buck 2011-11-23 14:44:34 UTC
Actually, Mazz, I thought I read that the agents should be shut down prior to upgrade, so that is what I did.

Comment 4 Robert Buck 2011-11-23 14:49:26 UTC
The doc said, "Stop all JBoss ON agents.". So perhaps this needs to be clarified in the context of upgrades???

Comment 5 John Mazzitelli 2011-11-23 14:52:03 UTC
specificallly, the docs say:

"In almost all production environments, you will want to start the agent as a background daemon process. On UNIX, this means you usually run the agent at boot time from init.d. On Windows, this means you install the agent as a Windows Service that automatically starts at boot time. ***The agent auto-update feature assumes you start your agent in this way.*** If you do not, and the agent auto-updates itself, the old agent running in the console will be shutdown and the new agent will be restarted as a background service ***if at all possible***"

Comment 6 Robert Buck 2011-11-23 15:17:01 UTC
The instructions state this:

"Not every step in this upgrade procedure applies to every JBoss Operations Network installation. Just run through the steps in order, and perform the ones necessary for your deployment.

...

"2. Stop all JBoss ON agents. ..."

So perhaps this should be clarified? If someone reads, "stop all agents", then they will do exactly that. And that is what I did.

Regarding a prior comment related to output redirection, that should not matter; if changing redirection affects fundamental behavior of the agent, then we're in trouble.

Comment 8 John Mazzitelli 2011-11-23 15:34:23 UTC
auto-updating should never involve shutting down the agent. That's the whole point of auto-update - you install the new server, the agents (still running) connect, find they need to upgrade, and do so automatically. shutting down the agent defeats auto-update.

sounds like the docs need more clarification.

Comment 9 John Mazzitelli 2011-11-23 15:36:19 UTC
(In reply to comment #7)
> Here is the 3.0 doc I was looking at:
> 
> http://docs.redhat.com/docs/en-US/JBoss_Operations_Network/3.0/html/Installation_Guide/upgrading.html#JON_Server_Server_Upgrade

ok, that's definitely the issue - we need to mention in this page (which is the SERVER upgrade page) that if they plan on using auto-update of the agents, they don't have to shut the agents down but the agents needed to be installed/prepared in such a way to allow it (i.e. point to the auto-update agent docs from the server upgrade docs)

Comment 10 Robert Buck 2011-11-23 15:36:34 UTC
Perhaps the answer would be to define _what_ "configured for automatic upgrades" means, and how to set that up?

Should that go into here:

http://docs.redhat.com/docs/en-US/JBoss_Operations_Network/3.0/html/Installation_Guide/Agent_Installation.html#installing-agent

With a link from the prior doc to here?