Bug 1127414 - WARN [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed to failover to another server
Summary: WARN [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {Agen...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Agent
Version: JON 3.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: DR02
: JON 3.3.0
Assignee: RHQ Project Maintainer
QA Contact: Mike Foley
URL:
Whiteboard:
: 1127388 1127389 1127391 1127406 1127407 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-06 20:25 UTC by Matt Mahoney
Modified: 2014-09-05 11:01 UTC (History)
2 users (show)

Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-09-05 11:01:10 UTC
Type: Bug


Attachments (Terms of Use)
agent.log (465.70 KB, text/plain)
2014-08-06 20:25 UTC, Matt Mahoney
no flags Details

Description Matt Mahoney 2014-08-06 20:25:03 UTC
Created attachment 924594 [details]
agent.log

Description of problem:

WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed to failover to another server. Cause: org.jboss.remoting.transport.http.WebServerError: <html><head><title>JBoss Web/7.4.8.Final-redhat-4 - JBWEB000064: Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 404 - /jboss-remoting-servlet-invoker/ServerInvokerServlet</h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u>/jboss-remoting-servlet-invoker/ServerInvokerServlet</u></p><p><b>JBWEB000069: description</b> <u>JBWEB000124: The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>JBoss Web/7.4.8.Final-redhat-4</h3></body></html>
WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed to failover to another server. Cause: org.jboss.remoting.transport.http.WebServerError: <html><head><title>JBoss Web/7.4.8.Final-redhat-4 - JBWEB000064: Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 503 - Server is not ready yet</h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u>Server is not ready yet</u></p><p><b>JBWEB000069: description</b> <u>JBWEB000148: The requested service is not currently available.</u></p><HR size="1" noshade="noshade"><h3>JBoss Web/7.4.8.Final-redhat-4</h3></body></html>
WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed to failover to another server. Cause: org.jboss.remoting.transport.http.WebServerError: <html><head><title>JBoss Web/7.4.8.Final-redhat-4 - JBWEB000064: Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 503 - Server is not ready yet</h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u>Server is not ready yet</u></p><p><b>JBWEB000069: description</b> <u>JBWEB000148: The requested service is not currently available.</u></p><HR size="1" noshade="noshade"><h3>JBoss Web/7.4.8.Final-redhat-4</h3></body></html>
WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.server-thinks-agent-is-down}The server thinks this agent is down. Will notify the server of up-to-date information when possible.
WARN  [RHQ Server Polling Thread] (enterprise.communications.command.client.ServerPollingThread)- {ServerPollingThread.server-offline}The server has gone offline; client has been told to stop sending commands
WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.FailoverFailureCallback)- {AgentMain.too-many-failover-attempts}Too many failover attempts have been made [1]. Exception that triggered the failover: [org.jboss.remoting.transport.http.WebServerError: <html><head><title>JBoss Web/7.4.8.Final-redhat-4 - JBWEB000064: Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 404 - /jboss-remoting-servlet-invoker/ServerInvokerServlet</h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u>/jboss-remoting-servlet-invoker/ServerInvokerServlet</u></p><p><b>JBWEB000069: description</b> <u>JBWEB000124: The requested resource is not available.</u></p><HR size="1" noshade="noshade"><h3>JBoss Web/7.4.8.Final-redhat-4</h3></body></html>]
WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.FailoverFailureCallback)- {AgentMain.too-many-failover-attempts}Too many failover attempts have been made [1]. Exception that triggered the failover: [org.jboss.remoting.transport.http.WebServerError: <html><head><title>JBoss Web/7.4.8.Final-redhat-4 - JBWEB000064: Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>JBWEB000065: HTTP Status 503 - Server is not ready yet</h1><HR size="1" noshade="noshade"><p><b>JBWEB000309: type</b> JBWEB000067: Status report</p><p><b>JBWEB000068: message</b> <u>Server is not ready yet</u></p><p><b>JBWEB000069: description</b> <u>JBWEB000148: The requested service is not currently available.</u></p><HR size="1" noshade="noshade"><h3>JBoss Web/7.4.8.Final-redhat-4</h3></body></html>]
WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.FailoverFailureCallback)- {AgentMain.too-many-failover-attempts}Too many failover attempts have been made [1]. Exception that triggered the failover: [org.jboss.remoting.CannotConnectException: Can not connect http client invoker after 1 attempt(s)]
WARN  [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.failover-failed}Failed to failover to another server. Cause: org.jboss.remoting.CannotConnectException: Can not connect http client invoker after 1 attempt(s)

Version-Release number of selected component (if applicable):
JON 3.3 DR01

How reproducible:


Steps to Reproduce:
1. Install and start JON.
2.
3.

Actual results:
Warnings in agent.log after clean install/start.

Expected results:
Should be zero silent Warnings in server.log

Additional info:

Comment 1 Heiko W. Rupp 2014-08-26 11:37:43 UTC
*** Bug 1127407 has been marked as a duplicate of this bug. ***

Comment 2 Heiko W. Rupp 2014-08-26 11:39:22 UTC
*** Bug 1127406 has been marked as a duplicate of this bug. ***

Comment 3 Heiko W. Rupp 2014-08-26 12:15:38 UTC
*** Bug 1127391 has been marked as a duplicate of this bug. ***

Comment 4 Heiko W. Rupp 2014-08-26 12:16:51 UTC
*** Bug 1127389 has been marked as a duplicate of this bug. ***

Comment 5 Heiko W. Rupp 2014-08-26 12:18:37 UTC
*** Bug 1127388 has been marked as a duplicate of this bug. ***

Comment 6 John Mazzitelli 2014-08-29 20:13:23 UTC
Is this really "high" severity? This is only log msgs saying the server isn't ready yet and the agent is trying to communicate with it but can't yet. I would suspect after just a short time, the agent will be able to communicate successfully, once the server finishes initializing and starts its comm layer.

I recommend lowering severity as this is only an issue with log messages and not really with actual behavior of the agent (unless I am mistaken - the description in this BZ does not mention if the agent never can connect to the server. If the agent is dead in the water and never connects, that would be a problem).

Comment 7 Lukas Krejci 2014-09-05 11:01:10 UTC
This is NOT a bug, this is the agent telling that it cannot (yet) communicate with the server. I'd consider the silence to be the actual bug here.

The agent has no way of telling whether the server is just starting up or if it somehow failed to start up, is in the process of shutting down or some other failure occurred on the server side. Logging such occurrence is the ONLY sensible thing to do on the agent side.

I would even consider it a bug if this was logged on DEBUG level, because diagnosing such error would require user/GSS intervention.

I'm closing this. Not every log message we emit is worth a BZ. If you think we should rephrase the log message or try to parse the error message coming from the server to try to extract some more meaning from it that we then somehow present to the user, you can reopen this BZ. I personally don't think it is worth the effort.


Note You need to log in before you can comment on or make changes to this bug.