Originating case: 313963 (has attachments displaying the symptomatic errors) Some environments suffer from the a series of potentially related Server log ERROR messages. The belief is that it may be tied to network instability. Although end user impact is minimal, the errors are disconcerting to the system manager, and perhaps there is some performance impact. Minimally, the errors should be better understood, and perhaps suppressed. There is also some indication that load and/or db locking may contribute. Large group manipulation may result in this situation. Here are examples: 2009-07-27 10:20:24,506 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure java.io.EOFException at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:526) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165) 2009-07-27 10:20:24,507 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#26[hidden:hidden] closing socketWrapper: ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].28b5982d] 2009-07-27 10:20:24,507 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000 2009-07-27 10:20:24,507 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure java.io.EOFException at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:526) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165) 2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.ServerThread] WorkerThread#182[hidden:hidden] closing socketWrapper: ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].264cba93] 2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] wrote CLOSING 2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].264cba93] closing 2009-07-27 10:20:24,508 DEBUG [org.jboss.remoting.transport.socket.ServerSocketWrapper] unable to writing CLOSING byte java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1784) at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:691) at org.jboss.remoting.transport.socket.ServerSocketWrapper.close(ServerSocketWrapper.java:57) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:481) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165) 2009-07-27 10:20:24,509 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] unable to close output stream 2009-07-27 10:20:24,509 DEBUG [org.jboss.remoting.transport.socket.SocketWrapper] ServerSocketWrapper[Socket[addr=/hidden,port=hidden,localport=hidden].28b5982d] closing 2009-07-27 10:16:31,715 DEBUG [org.jboss.remoting.transport.socket.ClientSocketWrapper] reset timeout: 60000 2009-07-02 11:03:22,286 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249) at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429) at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499) at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571) at java.io.ObjectInputStream.read(ObjectInputStream.java:820) at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:810) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:506) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165) 2009-07-02 11:04:17,136 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2249) at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2429) at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2499) at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2571) at java.io.ObjectInputStream.read(ObjectInputStream.java:820) at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:810) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:506) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:369) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
There is a tangentially related forum thread: http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4087180#4087180
Some similarity to messages seen here
Try to turn off multicast detection. I saw something yesterday where other jboss remoting servers were getting detected and the jboss/remoting stuff under the covers seemed to try to communicate with those endpoints. To turn off multicast detection, these preferences in the agent must be set to false: rhq.agent.server-auto-detection rhq.communications.multicast-detector.enabled (BTW: The default for these is now false - I made that change yesterday so future builds will have these disabled out of box)
Server-Side Multicast detection has also been set off by default in 2.3. It may be useful to turn off server side multicast in 2.2 to help avoid these errors. To do this: 1) shut down RHQ server 2) edit bin/rhq-server.properties set: rhq.communications.multicast-detector.enabled=false 3) restart server
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-2282 This bug is related to RHQ-1801
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs. keyword: new = Tracking + FutureFeature + SubBug
making sure we're not missing any bugs in rhq_triage
jay - could we turn detector off and close bug?
I believe multicast is now off by default in more recent versions. So, I think this can be closed.