Bug 601104 - Agent selfupdate failed
Summary: Agent selfupdate failed
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: RHQ Project
Classification: Other
Component: Agent
Version: 3.0.0
Hardware: All
OS: Linux
urgent
high
Target Milestone: ---
: ---
Assignee: John Sanda
QA Contact: Corey Welton
URL:
Whiteboard:
Depends On:
Blocks: jon-sprint11-bugs
TreeView+ depends on / blocked
 
Reported: 2010-06-07 08:47 UTC by Heiko W. Rupp
Modified: 2010-06-18 19:36 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-06-18 18:19:33 UTC
Embargoed:


Attachments (Terms of Use)
agent update log from windows 2003 server box (127.05 KB, application/octet-stream)
2010-06-18 17:45 UTC, John Sanda
no flags Details

Description Heiko W. Rupp 2010-06-07 08:47:25 UTC
In installed JON Beta 1 on a RHEL box where Beta0 was running successful for some time. I did not stop the agent for the install.
After the B1 server was up and running, the agent connected and the server told it that it is too old. Agent then tried to self update, which failed and I ended up with no agent running.

RHEL 5.5, sun jdk1.6, pg8.4

> This version of the agent is not supported by the server - an agent update must be applied
The agent update thread has started - will begin the agent auto-update now!
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
This version of the agent is not supported by the server - an agent update must be applied
The agent will wait for [0] threads to die
Downloaded the agent update binary to [/home/hrupp/rhq-enterprise-agent-3.0.0.Beta1.jar]
Missing the envvar [RHQ_AGENT_JAVA_EXE_FILE_PATH] - will try to find a Java executable to use.
Will use the Java executable [/usr/lib/jvm/java-1.6.0-sun-1.6.0.20/jre/bin/java]
Now executing agent update - if all goes well, this is the last you will hear of this agent: [ProcessExecution: executable=[/usr/lib/jvm/java-1.6.0-sun-1.6.0.20/jre/bin/java], args=[[-jar, /home/hrupp/rhq-enterprise-agent-3.0.0.Beta1.jar, --pause=20000, --update=/home/hrupp/rhq-agent]], env-vars=[null], working-dir=[/home/hrupp], wait=[0], capture-output=[false], kill-on-timeout=[false], executable-is-command=[true]]
[hrupp@pickeldi rhq-agent]$ jps
24376 Jps
26723 Main
9353 Main
23340 Main


server console:

10:28:35,362 INFO  [CoreServerServiceImpl] Agent [pickeldi][3.0.0.Beta0(a2b6884)] would like to connect to this server
10:28:35,365 WARN  [CoreServerServiceImpl] Agent [pickeldi][3.0.0.Beta0(a2b6884)] would like to connect to this server but it is not supported
10:28:35,769 INFO  [CoreServerServiceImpl] Agent [pickeldi][3.0.0.Beta0(a2b6884)] would like to connect to this server
10:28:35,772 WARN  [CoreServerServiceImpl] Agent [pickeldi][3.0.0.Beta0(a2b6884)] would like to connect to this server but it is not supported


I was also having agent upgrade issues with agent upgrades on OS X, but attributed it to OS X


----------


As the above was with an agent that was not started with the wrapper, I thought this may be the case, so I started it with the wrapper script and it started to try to upgrade, but it is now caught in a loop 

[hrupp@pickeldi ~]$ jps
25870 Jps
26723 Main
25711 rhq-enterprise-agent-3.0.0.Beta1.jar
9353 Main
23340 Main
25839 AgentMain
[hrupp@pickeldi ~]$ jps
25918 Jps
26723 Main
25895 rhq-enterprise-agent-3.0.0.Beta1.jar
9353 Main
23340 Main
[hrupp@pickeldi ~]$ jps
26723 Main
26223 Jps
9353 Main
26196 rhq-enterprise-agent-3.0.0.Beta1.jar
23340 Main

So it connects to the server downloads the bits, does not manage to install them, connects again with the old version ...

Server log:

10:46:44,010 ERROR [[/]] AgentUpdateServlet: Failed to stream agent jar
java.lang.RuntimeException: Stream data cannot be copied
        at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:118)
        at org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet.getDownload(AgentUpdateServlet.java:131)
        at org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet.doGet(AgentUpdateServlet.java:94)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:619)
Caused by: ClientAbortException:  java.net.SocketException: Connection reset
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:381)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:370)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
        at org.rhq.core.util.stream.StreamUtil.copy(StreamUtil.java:112)
        ... 25 more
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:737)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
        at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:761)
        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:118)
        at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
        at org.apache.coyote.Response.doWrite(Response.java:560)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
        ... 30 more

Comment 1 Heiko W. Rupp 2010-06-07 08:57:35 UTC
A manual update of the downloaded agent bits via java -jar rhq-enterprise-agent-..Beta1.jar --update works as expected.

Comment 2 John Mazzitelli 2010-06-18 13:51:06 UTC
if you expect auto-update to work, make sure you follow the requirements here:

http://rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-PreparingYourAgentToBeAutoUpdatable

If you don't do everything there, its possible the auto-update won't fully complete.

Comment 3 John Sanda 2010-06-18 17:45:10 UTC
Created attachment 425202 [details]
agent update log from windows 2003 server box

Comment 4 John Sanda 2010-06-18 18:19:33 UTC
This time when I made sure not to have any terminals open in the directories being deleted, the agent self updated succeeded without error on my Windows 2003 instance. The agent restarted on its own without issue. I am going to close this bug out.

Comment 5 John Sanda 2010-06-18 19:36:31 UTC
Looks like my comments prior to comment 4 got lost when I attached the log file. Using the docs referenced by mazz in comment 1 I tested upgrading from JON 2.3.1 to 2.4.0. I tested the agent update on Mac OS X, Fedora 12, and on Windows 2003 server. The auto-update completed without error on Mac OS X and on F12. I ran into a problem on Windows though which you can see in the log I attached in comment 3.

Due to Windows file locking, the updater could not delete rhq-agent/bin which resulted in the agent failing to restart. I tested a subsequent time on windows making sure I didn't have any terminals open in the directories being deleted. This time the update completed without error.


Note You need to log in before you can comment on or make changes to this bug.