Bug 790152

Summary: JON 3.01 RC4, rhq-agent displays remote client exception
Product: [Other] RHQ Project Reporter: Mike Foley <mfoley>
Component: AgentAssignee: John Mazzitelli <mazz>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Mike Foley <mfoley>
Severity: unspecified Docs Contact:
Priority: high    
Version: 3.0.1CC: hrupp, jshaughn, mazz
Target Milestone: ---   
Target Release: JON 3.1.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-14 18:23:47 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
agent log
none
change to remove printStackTrace none

Description Mike Foley 2012-02-13 19:23:10 UTC 
Created attachment 561648 [details]
agent log

Description of problem:  JON 3.01 RC4, rhq-agent displays remote client exception, as shown below.  I have no idea how to repro this or how it happened....hopefully the stack trace will yield some clues.    Attaching agent log.





UP:	CPU 3
Sending the availability report to the server...
Done.
> org.rhq.enterprise.communications.command.client.RemoteIOException: A remote input stream with an ID of [4] and server endpoint of [socket://192.168.0.103:16163/?backlog=200&clientMaxPoolSize=304&enableTcpNoDelay=true&maxPoolSize=303&numAcceptThreads=1&rhq.communications.connector.rhqtype=agent&socketTimeout=60000] has not yet been assigned a sender object - cannot access the stream
	at org.rhq.enterprise.communications.command.client.RemoteInputStream.sendRequest(RemoteInputStream.java:266)
	at org.rhq.enterprise.communications.command.client.RemoteInputStream.close(RemoteInputStream.java:186)
	at org.rhq.core.pc.content.RetrieveContentBitsRunner.run(RetrieveContentBitsRunner.java:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)

>
Comment 1 Mike Foley 2012-02-27 16:35:10 UTC 
triage crouch, mfoley, loleary.

dev to take 1st pass.
Comment 2 John Mazzitelli 2012-02-27 17:01:28 UTC 
I really don't understand this code. Not sure why it is streaming the same file three times in a row.

The exception is dumped because someone used e.printStackTrace rather than doing what we are supposed to be doing in the PC - that is, logging the exception. I'll change that real quick, but that won't solve the issue - all it would do is move the excepton to the log file as opposed to the console.
Comment 3 John Mazzitelli 2012-02-27 17:09:44 UTC 
Created attachment 566104 [details]
change to remove printStackTrace

attached is a diff that removes the printStackTrace calls and replaces them with log calls.
Comment 4 John Mazzitelli 2012-02-27 17:14:01 UTC 
it looks like this is a failure while trying to calculate the SHA256 (athough I'm not sure why its doing this with a remote file).

Did this happen when you enabled content discovery on the resource?

The only other thing I can think of is that something got nicked durign the latest refactoring of the content subsystem.
Comment 5 John Mazzitelli 2012-02-27 17:19:36 UTC 
for the record, git commit c7a3cd80a75f9f5141a5a95553906eb4e361d134 added these additional two streaming calls to set the sha256 and md5. Not sure why it is being asked to stream three separate times - we could probably have streamed it once and done all three things we needed concurrently. But in any event, this was introduced close to 2 years ago, so somethng else must have nicked this and caused the streaming to now break (??)
Comment 6 John Mazzitelli 2012-02-27 19:44:43 UTC 
git commit to master: 4daf76814cf38969fb1873938b78a43d3119aaf6
Comment 7 John Mazzitelli 2012-02-27 19:45:18 UTC 
(In reply to comment #6)
> git commit to master: 4daf76814cf38969fb1873938b78a43d3119aaf6

just to be clear, that commit ONLY removes the printStackTrace and logs the error instead. It doesn't attempt to fix anything.
Comment 10 Charles Crouch 2012-08-03 02:17:44 UTC 
Mazz, can you take another look at this and see if there is anything that jumps out as could be causing this.
Comment 11 John Mazzitelli 2012-08-03 14:18:22 UTC 
as an aside and for the record - 4daf76814cf38969fb1873938b78a43d3119aaf6 is already in the release/jon3.1.x branch.
Comment 12 John Mazzitelli 2012-08-14 18:23:47 UTC 
no way to know what happened. If the server was in the process of shutting down, this could be an expected condition. Since it was a one-off anomoly and can't be reproduced, closing wiht INSUFFICIENT_DATA. We can revisit if this happens again. But again, this is expected if the server was in the middle of shutting down while the agent is streaming.