Bug 535604 - (RHQ-2282) Server log errors: Worker Thread Initialization Failure error, broken pipe, connection reset
Server log errors: Worker Thread Initialization Failure error, broken pipe, c...
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: No Component (Show other bugs)
1.2
All All
medium Severity medium (vote)
: ---
: ---
Assigned To: RHQ Project Maintainer
http://jira.rhq-project.org/browse/RH...
: SubBug
Depends On:
Blocks: rhq_triage
  Show dependency treegraph
 
Reported: 2009-07-31 10:39 EDT by Jay Shaughnessy
Modified: 2014-05-09 11:28 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
The same JDK version server/agent : java version "1.6.0_13" Oracle I think these errors are OS independent
Last Closed: 2014-05-09 11:28:36 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jay Shaughnessy 2009-07-31 10:39:00 EDT
Originating case: 313963 (has attachments displaying the symptomatic errors)

Some environments suffer from the a series of potentially related Server log ERROR messages.  The belief is that it may be tied to network instability.  Although end user impact is minimal, the errors are disconcerting to the system manager, and perhaps there is some performance impact.  Minimally, the errors should be better understood, and perhaps suppressed.

There is also some indication that load and/or db locking may contribute.  Large group manipulation may result in this situation.

Here are examples:

2009-07-27 10:20:24,506 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.io.EOFException
	at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:526)
	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369)
	at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
2009-07-27 10:20:24,507 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#26[hidden:hidden] closing socketWrapper: ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].28b5982d]
2009-07-27 10:20:24,507 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000
2009-07-27 10:20:24,507 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.io.EOFException
	at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:526)
	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369)
	at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#182[hidden:hidden] closing socketWrapper: ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].264cba93]
2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] wrote CLOSING
2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].264cba93] closing
2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] unable to writing CLOSING byte
java.net.SocketException: Broken pipe
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
	at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1784)
	at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:691)
	at org.jboss.remoting.transport.socket.ServerSocketWrapper.close(ServerSocketWrapper.java:57)
	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:481)
	at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
2009-07-27 10:20:24,509 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] unable to close output stream
2009-07-27 10:20:24,509 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].28b5982d] closing
2009-07-27 10:16:31,715 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000


2009-07-02 11:03:22,286 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
        at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
        at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
        at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
        at java.io.ObjectInputStream.read(ObjectInputStream.java:820)
        at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:810)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:506)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)


2009-07-02 11:04:17,136 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249)
        at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429)
        at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499)
        at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571)
        at java.io.ObjectInputStream.read(ObjectInputStream.java:820)
        at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:810)
        at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:506)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)


Comment 1 Jay Shaughnessy 2009-07-31 10:40:24 EDT
There is a tangentially related forum thread: http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4087180#4087180
Comment 2 Jay Shaughnessy 2009-07-31 10:41:06 EDT
Some similarity to messages seen here
Comment 3 John Mazzitelli 2009-07-31 10:57:17 EDT
Try to turn off multicast detection. I saw something yesterday where other jboss remoting servers were getting detected and the jboss/remoting stuff under the covers seemed to try to communicate with those endpoints.

To turn off multicast detection, these preferences in the agent must be set to false:

rhq.agent.server-auto-detection
rhq.communications.multicast-detector.enabled

(BTW: The default for these is now false - I made that change yesterday so future builds will have these disabled out of box)
Comment 4 Jay Shaughnessy 2009-07-31 13:45:49 EDT
Server-Side Multicast detection has also been set off by default in 2.3.  

It may be useful to turn off server side multicast in 2.2 to help avoid these errors.  To do this:

1) shut down RHQ server
2) edit bin/rhq-server.properties set:

rhq.communications.multicast-detector.enabled=false

3) restart server



Comment 5 Red Hat Bugzilla 2009-11-10 16:01:25 EST
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-2282
This bug is related to RHQ-1801
Comment 6 wes hayutin 2010-02-16 11:53:30 EST
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs.

keyword:
new = Tracking + FutureFeature + SubBug
Comment 7 wes hayutin 2010-02-16 11:58:50 EST
making sure we're not missing any bugs in rhq_triage
Comment 8 Corey Welton 2010-11-22 14:38:26 EST
jay - could we turn detector off and close bug?
Comment 9 Jay Shaughnessy 2010-11-23 09:11:04 EST
I believe multicast is now off by default in more recent versions. So, I think this can be closed.

Note You need to log in before you can comment on or make changes to this bug.