Bug 1377310 - Engine log is flooded by UnknownHostException logs
Summary: Engine log is flooded by UnknownHostException logs
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Infra
Version: 4.0.4.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.0.5
: 4.0.5
Assignee: Martin Perina
QA Contact: Petr Kubica
URL:
Whiteboard:
Depends On:
Blocks: 1390192
TreeView+ depends on / blocked
 
Reported: 2016-09-19 12:21 UTC by Yaniv Kaul
Modified: 2017-01-18 07:39 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-01-18 07:39:40 UTC
oVirt Team: Infra
Embargoed:
ykaul: ovirt-4.0.z?
rule-engine: planning_ack?
mperina: devel_ack+
pstehlik: testing_ack+


Attachments (Terms of Use)
engine.log (2.39 MB, application/x-gzip)
2016-09-19 12:21 UTC, Yaniv Kaul
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 64285 0 master NEW core: Improve error logging in HostMonitoring 2016-09-22 10:52:33 UTC

Description Yaniv Kaul 2016-09-19 12:21:16 UTC
Created attachment 1202468 [details]
engine.log

Description of problem:
Looking at RHEV.TLV setup, there is a flood of logs due to exception with monster01. It keeps repeating often.
It's enough that 1-2 hosts are not responding (out of many!) and the logs are quite flooded.

Version-Release number of selected component (if applicable):
ovirt-engine-4.0.4.2-0.1.el7ev.noarch


2016-09-19 15:16:43,405 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler55) [46908fbb] Failed to fetch vms info for host 'monster01' - skipping VMs monitoring.
2016-09-19 15:16:43,886 WARN  [org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable] (SSL Stomp Reactor) [] Retry failed
2016-09-19 15:16:43,887 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (DefaultQuartzScheduler11) [652074d9] Exception during connection
2016-09-19 15:16:43,889 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler11) [652074d9] Command 'GetCapabilitiesVDSCommand(HostName = monster01, VdsIdAndVdsVDSCo
mmandParametersBase:{runAsync='true', hostId='4d81dd2e-e5ca-441c-a700-0c72b771c1c2', vds='Host[monster01,4d81dd2e-e5ca-441c-a700-0c72b771c1c2]'})' execution failed: java.net.UnknownHostException: monster01.eng.l
ab.tlv.redhat.com
2016-09-19 15:16:43,889 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler11) [652074d9] Failure to refresh Vds runtime info: java.net.UnknownHostException: monster01.eng.l
ab.tlv.redhat.com
2016-09-19 15:16:43,889 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler11) [652074d9] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.
UnknownHostException: monster01.eng.lab.tlv.redhat.com
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:157) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:120) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]
        at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:653) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:121) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:85) [vdsbroker.jar:]
        at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238) [vdsbroker.jar:]
        at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) [:1.8.0_101]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101]
        at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101]
        at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:]
        at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]
Caused by: java.net.UnknownHostException: monster01.eng.lab.tlv.redhat.com
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280) [rt.jar:1.8.0_101]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) [rt.jar:1.8.0_101]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) [rt.jar:1.8.0_101]
        at java.net.InetAddress.getByName(InetAddress.java:1076) [rt.jar:1.8.0_101]
        at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:121) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient$1.call(ReactorClient.java:117) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.utils.retry.Retryable.call(Retryable.java:27) [vdsm-jsonrpc-java-client.jar:]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101]
        at org.ovirt.vdsm.jsonrpc.client.utils.ReactorScheduler.performPendingOperations(ReactorScheduler.java:28) [vdsm-jsonrpc-java-client.jar:]
        at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.run(Reactor.java:61) [vdsm-jsonrpc-java-client.jar:]

Comment 1 Oved Ourfali 2016-09-19 12:24:37 UTC
It probably stops after identifying that the host isn't reachable.
Isn't it?

Comment 2 Yaniv Kaul 2016-09-19 12:40:15 UTC
(In reply to Oved Ourfali from comment #1)
> It probably stops after identifying that the host isn't reachable.
> Isn't it?

No:

...
2016-09-19 15:39:15,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler21) [] Command 'GetCapabilitiesVDSCommand(HostName = monster01, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='4d81dd2e-e5ca-441c-a700-0c72b771c1c2', vds='Host[monster01,4d81dd2e-e5ca-441c-a700-0c72b771c1c2]'})' execution failed: java.net.UnknownHostException: monster01.eng.lab.tlv.redhat.com: unknown error
2016-09-19 15:39:18,960 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler11) [652074d9] Command 'GetCapabilitiesVDSCommand(HostName = monster01, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='4d81dd2e-e5ca-441c-a700-0c72b771c1c2', vds='Host[monster01,4d81dd2e-e5ca-441c-a700-0c72b771c1c2]'})' execution failed: java.net.UnknownHostException: monster01.eng.lab.tlv.redhat.com
2016-09-19 15:39:21,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [] Command 'GetCapabilitiesVDSCommand(HostName = monster01, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='4d81dd2e-e5ca-441c-a700-0c72b771c1c2', vds='Host[monster01,4d81dd2e-e5ca-441c-a700-0c72b771c1c2]'})' execution failed: java.net.UnknownHostException: monster01.eng.lab.tlv.redhat.com
2016-09-19 15:39:24,988 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler30) [69f21400] Command 'GetCapabilitiesVDSCommand(HostName = monster01, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='4d81dd2e-e5ca-441c-a700-0c72b771c1c2', vds='Host[monster01,4d81dd2e-e5ca-441c-a700-0c72b771c1c2]'})' execution failed: java.net.UnknownHostException: monster01.eng.lab.tlv.redhat.com
...

[ykaul@nott16 ovirt-engine]$ grep -c "GetCapabilitiesVDSCommand(HostName = monster01" engine.log
12644

Comment 3 Oved Ourfali 2016-09-19 12:50:14 UTC
Martin - let's add that to the logging items list.

Comment 4 Petr Kubica 2016-10-19 13:30:52 UTC
Verified in rhevm-4.0.5.1-0.1.el7ev.noarch


Note You need to log in before you can comment on or make changes to this bug.